Abstract:Open-vocabulary learning has emerged as a cutting-edge research area, particularly in light of the widespread adoption of vision-based foundational models. Its primary objective is to comprehend novel concepts that are not encompassed within a predefined vocabulary. One key facet of this endeavor is Visual Grounding (VG), which entails locating a specific region within an image based on a corresponding language description. While current foundational models excel at various visual language tasks, there's a noticeable absence of models specifically tailored for open-vocabulary visual grounding (OV-VG). This research endeavor introduces novel and challenging OV tasks, namely Open-Vocabulary Visual Grounding (OV-VG) and Open-Vocabulary Phrase Localization (OV-PL). The overarching aim is to establish connections between language descriptions and the localization of novel objects. To facilitate this, we have curated a comprehensive annotated benchmark, encompassing 7,272 OV-VG images (comprising 10,000 instances) and 1,000 OV-PL images. In our pursuit of addressing these challenges, we delved into various baseline methodologies rooted in existing open-vocabulary object detection (OV-D), VG, and phrase localization (PL) frameworks. Surprisingly, we discovered that state-of-the-art (SOTA) methods often falter in diverse scenarios. Consequently, we developed a novel framework that integrates two critical components: Text-Image Query Selection (TIQS) and Language-Guided Feature Attention (LGFA). These modules are designed to bolster the recognition of novel categories and enhance the alignment between visual and linguistic information. Extensive experiments demonstrate the efficacy of our proposed framework, which consistently attains SOTA performance across the OV-VG task. Additionally, ablation studies provide further evidence of the effectiveness of our innovative models. Codes and datasets will be made publicly available at https://github.com/cv516Buaa/OV-VG .

Towards Vocabulary-Independent Speech Indexing for Large-Scale Repositories

Addressing the Out-of-vocabulary Problem for Large-Scale Chinese Spoken Term Detection

Out-of-Vocabulary Issue in Chinese Spoken Term Detection and A Two-Stage Chinese Speech Retrieval Method

Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition

AudioVSR: Enhancing Video Speech Recognition with Audio Data

Learning to Distribute Vocabulary Indexing for Scalable Visual Search

Open-Vocabulary Retrieval of Spoken Content with Shorter/Longer Queries Considering Word/Subword-based Acoustic Feature Similarity.

Enhancing Out-of-Vocabulary Performance of Indian TTS Systems for Practical Applications through Low-Effort Data Strategies

Using Word Confidence Measure for OOV Words Detection in a Spontaneous Spoken Dialog System.

Combining Lattice-Based Language Dependent and Independent Approaches for Out-of-Language Detection in LVCSR.

Modeling Image Data for Effective Indexing and Retrieval in Large General Image Databases.

Scalable Identity-Oriented Speech Retrieval

Learning Task-Specific Representation for Novel Words in Sequence Labeling.

Prediction of Neutrino Fluxes in the NOMAD Experiment

OV-VG: A benchmark for open-vocabulary visual grounding

Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models

A Unified Model for Solving the OOV Problem of Chinese Word Segmentation

Federated Learning Of Out-Of-Vocabulary Words

Improving Out-of-Vocabulary Handling in Recommendation Systems

Evaluating Inverted Files for Visual Compact Codes on a Large Scale

Incorporate Web Search Technology to Solve Out-of-Vocabulary Words in Chinese Word Segmentation.