Abstract:Visual grounding (VG) is a task that requires to locate a specific region in an image according to a natural language expression. Existing efforts on the VG task are divided into two-stage, one-stage and Transformer-based methods, which have achieved high performance. However, most of the previous methods extract visual information at a single spatial scale and ignore visual information at other spatial scales, which makes these models unable to fully utilize the visual information. Moreover, the insufficient utilization of linguistic information, especially failure to capture global linguistic information, may lead to failure to fully understand language expressions, thus limiting the performance of these models. To better address the task, we propose a language conditioned multi-scale visual attention network (LMSVA) for visual grounding, which can sufficiently utilize visual and linguistic information to perform multimodal reasoning, thus improving performance of model. Specifically, we design a visual feature extractor containing a multi-scale layer to get the required multi-scale visual features by expanding the original backbone. Moreover, we exploit pooling the output of the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to extract sentence-level linguistic features, which can enable the model to capture global linguistic information. Inspired by the Transformer architecture, we present the Visual Attention Layer guided by Language and Multi-Scale Visual Features (VALMS), which is able to better learn the visual context guided by multi-scale visual and linguistic features, and facilitates further multimodal reasoning. Extensive experiments on four large benchmark datasets, including ReferItGame, RefCOCO, RefCOCO+ and RefCOCOg, demonstrate that our proposed model achieves the state-of-the-art performance.

Learning to Assemble Neural Module Tree Networks for Visual Grounding.

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention

Visual-Semantic Graph Matching for Visual Grounding

Learning to Compose and Reason with Language Tree Structures for Visual Grounding.

Visualizing and Understanding Neural Models in NLP

Recursive Grounding Pruning Input Language the skis of the man in the red jacket skis of man in red jacket RvG-Tree Constructor

Language-guided Visual Attention Network for Visual Grounding

Visual Question Reasoning on General Dependency Tree

Joint Visual Grounding with Language Scene Graphs

Interpretable Visual Question Answering by Reasoning on Dependency Trees

Language Conditioned Multi-Scale Visual Attention Networks for Visual Grounding

End-to-End Modeling Via Information Tree for One-Shot Natural Language Spatial Video Grounding

Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding

vtGraphNet: Learning weakly-supervised scene graph for complex visual grounding

Learning to Reason: End-to-End Module Networks for Visual Question Answering

End-to-end Visual Grounding Based on Query Text Guidance and Multi-stage Reasoning

Multi-Granularity Modularized Network for Abstract Visual Reasoning

From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

Transformer-based Visual Grounding with Cross-modality Interaction

Visual Grounding With Joint Multimodal Representation and Interaction