Abstract:Visual Question Answering (VQA) is a task that requires VQA model to fully understand the visual information of the image and the language information of the question, and then combine both to provide an answer. Recently, a large amount of VQA approaches focus on modeling intra- and inter-modal interactions with respect to vision and language using a deep modular co-attention network, which can achieve a good performance. Despite their benefits, they also have their limitations. First, the question representation is obtained through Glove word embeddings and Recurrent Neural Network, which may not be sufficient to capture the intricate semantics of the question features. Second, they mostly use visual appearance features extracted by Faster R-CNN to interact with language features, and they ignore important spatial relations between objects in images, resulting in incomplete use of image information. To overcome the limitations of previous methods, we propose a novel Multi-modal Spatial Relation Attention Network (MSRAN) for VQA, which can introduce spatial relationships between objects to fully utilize the image information, thus improving the performance of VQA. In order to achieve the above, we design two types of spatial relational attention modules to comprehensively explore the attention schemes: (i) Self-Attention based on Explicit Spatial Relation (SA-ESR) module that explores geometric relationships between objects explicitly; and (ii) Self-Attention based on Implicit Spatial Relation (SA-ISR) module that can capture the hidden dynamic relationships between objects by using spatial relationship. Moreover, the pre-training model BERT, which replaces Glove word embeddings and Recurrent Neural Network, is applied to MSRAN in order to obtain the better question representation. Extensive experiments on two large benchmark datasets, VQA 2.0 and GQA, demonstrate that our proposed model achieves the state-of-the-art performance.

Weakly-Supervised 3D Spatial Reasoning for Text-based Visual Question Answering

Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering

Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA

An effective spatial relational reasoning networks for visual question answering

Learning Hierarchical Reasoning for Text-Based Visual Question Answering

CLEVR3D: Compositional Language and Elementary Visual Reasoning for Question Answering in 3D Real-World Scenes

3D-Aware Visual Question Answering about Parts, Poses and Occlusions

Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Multi-modal Spatial Relational Attention Networks for Visual Question Answering

SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities

Spatially aware multimodal transformers for textvqa

3D Question Answering with Scene Graph Reasoning

Two-Stage Multimodality Fusion for High-Performance Text-Based Visual Question Answering.

Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA

Generating Visual Spatial Description via Holistic 3D Scene Understanding

3D visual question answering based on sub-questions asymptotic reasoning

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Improving Vision-and-Language Reasoning via Spatial Relations Modeling

Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering