Abstract:Visual Question Answering (VQA) is a method of answering questions in natural language based on the content of images and has been widely concerned by researchers. The existing research on the visual question answering model mainly focuses on the point of view of attention mechanism and multi-modal fusion. It only pays attention to the visual semantic features of the image in the process of image modeling, ignoring the importance of modeling the spatial relationship of visual objects. We are aiming at the existing problems of the existing VQA model research. An effective spatial relationship reasoning network model is proposed, which can combine visual object semantic reasoning and spatial relationship reasoning at the same time to realize fine-grained multi-modal reasoning and fusion. A sparse attention encoder is designed to capture contextual information effectively in the semantic reasoning module. In the spatial relationship reasoning module, the graph neural network attention mechanism is used to model the spatial relationship of visual objects, which can correctly answer complex spatial relationship reasoning questions. Finally, a practical compact self-attention (CSA) mechanism is designed to reduce the redundancy of self-attention in linear transformation and the number of model parameters and effectively improve the model's overall performance. Quantitative and qualitative experiments are conducted on the benchmark datasets of VQA 2.0 and GQA. The experimental results demonstrate that the proposed method performs favorably against the state-of-the-art approaches. Our best single model has an overall accuracy of 71.18% on the VQA 2.0 dataset and 57.59% on the GQA dataset.

Scene Graph Refinement Network for Visual Question Answering

Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering

An Empirical Study on Leveraging Scene Graphs for Visual Question Answering

Understanding the Role of Scene Graphs in Visual Question Answering

Prior Visual Relationship Reasoning For Visual Question Answering

Graph Reasoning Networks for Visual Question Answering

Graph-Structured Representations for Visual Question Answering

SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering

Question-relationship guided graph attention network for visual question answer

Joint Learning of Object Graph and Relation Graph for Visual Question Answering

DSGEM: Dual Scene Graph Enhancement Module‐based Visual Question Answering

SceneGATE: Scene-Graph Based Co-Attention Networks for Text Visual Question Answering

Question-aware Dynamic Scene Graph of Local Semantic Representation Learning for Visual Question Answering

Visual Graph Reasoning Network

Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering

Question-Driven Graph Fusion Network For Visual Question Answering

An effective spatial relational reasoning networks for visual question answering

Visual Question Answering by Pattern Matching and Reasoning

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning

Co-attention graph convolutional network for visual question answering