Abstract:This paper tackles the challenging yet important task of Visual Grounding (VG), which aims to localize a visual region in the given image referred by a natural language query. Existing efforts on the VG task are twofold: 1) two-stage methods first extract region proposals and then rank them according to their similarities with the referring expression, which usually leads to suboptimal results due to the quality of region proposals; 2) one-stage methods usually predict all the possible coordinates of the target region online by leveraging modern object detection architectures, which pay few attention to cross-modality correlations and have limited generalization ability. To better address the task, we present an effective transformer-based end-to-end visual grounding approach, which focuses on capturing the cross-modality correlations between the referring expression and visual regions for accurately reasoning the location of the target region. Specifically, our model consists of a feature encoder, a cross-modality interactor, and a modality-agnostic decoder. The feature encoder is employed to capture the intra-modality correlation, which models the linguistic context in query and the spatial dependency in image respectively. The cross-modality interactor endows the model with the capability of highlighting the localization-relevant visual and textual cues by mutual verification of vision and language, which plays a key role in our model. The decoder learns a consolidated token representation enriched by multi-modal contexts and further directly predicts the box coordinates. Extensive experiments on five public benchmark datasets with quantitative and qualitative analysis clearly demonstrate the effectiveness and rationale of our proposed method.

Cross-Modal Omni Interaction Modeling for Phrase Grounding

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics.

Interaction is Worth More Explanations: Improving Human-Object Interaction Representation with Propositional Knowledge

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

Visual Grounding With Joint Multimodal Representation and Interaction

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Multimodal Co-Attention Mechanism for One-stage Visual Grounding.

Hierarchical Cross-Modal Contextual Attention Network for Visual Grounding

Cross-Modal Relationship Inference for Grounding Referring Expressions

Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding

Weakly Supervised Multimodal Affordance Grounding for Egocentric Images

Omni-IML: Towards Unified Image Manipulation Localization

Transformer-based Visual Grounding with Cross-modality Interaction

Cross-Modal Match for Language Conditioned 3D Object Grounding

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection

Extending Phrase Grounding with Pronouns in Visual Dialogues

OmniBench: Towards The Future of Universal Omni-Language Models

A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models

From Specific-MLLM to Omni-MLLM: A Survey about the MLLMs alligned with Multi-Modality

Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding