Abstract:Open-vocabulary learning has emerged as a cutting-edge research area, particularly in light of the widespread adoption of vision-based foundational models. Its primary objective is to comprehend novel concepts that are not encompassed within a predefined vocabulary. One key facet of this endeavor is Visual Grounding (VG), which entails locating a specific region within an image based on a corresponding language description. While current foundational models excel at various visual language tasks, there's a noticeable absence of models specifically tailored for open-vocabulary visual grounding (OV-VG). This research endeavor introduces novel and challenging OV tasks, namely Open-Vocabulary Visual Grounding (OV-VG) and Open-Vocabulary Phrase Localization (OV-PL). The overarching aim is to establish connections between language descriptions and the localization of novel objects. To facilitate this, we have curated a comprehensive annotated benchmark, encompassing 7,272 OV-VG images (comprising 10,000 instances) and 1,000 OV-PL images. In our pursuit of addressing these challenges, we delved into various baseline methodologies rooted in existing open-vocabulary object detection (OV-D), VG, and phrase localization (PL) frameworks. Surprisingly, we discovered that state-of-the-art (SOTA) methods often falter in diverse scenarios. Consequently, we developed a novel framework that integrates two critical components: Text-Image Query Selection (TIQS) and Language-Guided Feature Attention (LGFA). These modules are designed to bolster the recognition of novel categories and enhance the alignment between visual and linguistic information. Extensive experiments demonstrate the efficacy of our proposed framework, which consistently attains SOTA performance across the OV-VG task. Additionally, ablation studies provide further evidence of the effectiveness of our innovative models. Codes and datasets will be made publicly available at https://github.com/cv516Buaa/OV-VG .

Revisiting Image-Language Networks for Open-Ended Phrase Detection

Open-vocabulary Phrase Detection

Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues

Hybrid Deep Neural Network for Visual Phrase Detection

Conditional Image-Text Embedding Networks

Phrase Localization Without Paired Training Examples

Align2ground: Weakly supervised phrase grounding guided by image-caption alignment

PIRC Net : Using Proposal Indexing, Relationships and Context for Phrase Grounding

Cross-Modal Omni Interaction Modeling for Phrase Grounding

G3raphGround: Graph-Based Language Grounding.

Propagating Over Phrase Relations for One-Stage Visual Grounding

Extending Phrase Grounding with Pronouns in Visual Dialogues

Adapting CLIP For Phrase Localization Without Further Training

Natural Language Inference Improves Compositionality in Vision-Language Models

Grounding Referring Expressions in Images by Variational Context

A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models

OV-VG: A benchmark for open-vocabulary visual grounding

Phrase Grounding by Soft-Label Chain Conditional Random Field

Learning Two-Branch Neural Networks for Image-Text Matching Tasks.

Discriminative Learning of Open-Vocabulary Object Retrieval and Localization by Negative Phrase Augmentation

Vision-by-Language for Training-Free Compositional Image Retrieval