Abstract:Visual grounding is a task of locating the object referred by a natural language description. To reduce annotation costs, recent researchers are devoted into one-stage weakly supervised methods for visual grounding, which typically adopt the anchor-text matching paradigm. Despite the efficiency, we identify that anchor representations are often noisy and insufficient to describe object information, which inevitably hinders the vision-language alignments. In this paper, we propose a novel query-based one-stage framework for weakly supervised visual grounding, namely QueryMatch. Different from previous work, QueryMatch represents candidate objects with a set of query features, which inherently establish accurate one-to-one associations with visual objects. In this case, QueryMatch re-formulates weakly supervised visual grounding as a query-text matching problem, which can be optimized via the query-based contrastive learning. Based on QueryMatch, we further propose an innovative strategy for effective weakly supervised learning, namely Active Query Selection (AQS). In particular, AQS aims to enhance the effectiveness of query-based contrastive learning by actively selecting high-quality query features. Through this strategy, AQS can greatly benefit the weakly supervised learning of QueryMatch. To validate our approach, we conduct extensive experiments on three benchmark datasets of two grounding tasks, i.e., referring expression comprehension (REC) and segmentation (RES). Experimental results not only show the state-of-art performance of QueryMatch in two tasks, e.g., over +5% [email protected] on RefCOCO in REC and over +20% mIOU on RefCOCO in RES, but also confirm the effectiveness of AQS in weakly supervised learning. Source codes are available at https://github.com/TensorThinker/QueryMatch.

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Visual-Semantic Graph Matching for Visual Grounding

Cross-Modal Match for Language Conditioned 3D Object Grounding

Visual Grounding With Joint Multimodal Representation and Interaction

Co-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videos

Cross-Modal Relationship Inference for Grounding Referring Expressions

Bottom-Up and Bidirectional Alignment for Referring Expression Comprehension.

Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing

SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attention

Referencing Where to Focus: Improving VisualGrounding with Referential Query

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

Relationship-Embedded Representation Learning for Grounding Referring Expressions

Bi-Directional Relationship Inferring Network For Referring Image Segmentation

Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding

QueryMatch: A Query-based Contrastive Learning Framework for Weakly Supervised Visual Grounding

Multimodal Co-Attention Mechanism for One-stage Visual Grounding.

Cross-Modal Omni Interaction Modeling for Phrase Grounding

Position-Aware Contrastive Alignment for Referring Image Segmentation

Bridging Modality Gap for Visual Grounding with Effecitve Cross-modal Distillation