Abstract:Visual Question Answering (VQA) is a method of answering questions in natural language based on the content of images and has been widely concerned by researchers. The existing research on the visual question answering model mainly focuses on the point of view of attention mechanism and multi-modal fusion. It only pays attention to the visual semantic features of the image in the process of image modeling, ignoring the importance of modeling the spatial relationship of visual objects. We are aiming at the existing problems of the existing VQA model research. An effective spatial relationship reasoning network model is proposed, which can combine visual object semantic reasoning and spatial relationship reasoning at the same time to realize fine-grained multi-modal reasoning and fusion. A sparse attention encoder is designed to capture contextual information effectively in the semantic reasoning module. In the spatial relationship reasoning module, the graph neural network attention mechanism is used to model the spatial relationship of visual objects, which can correctly answer complex spatial relationship reasoning questions. Finally, a practical compact self-attention (CSA) mechanism is designed to reduce the redundancy of self-attention in linear transformation and the number of model parameters and effectively improve the model's overall performance. Quantitative and qualitative experiments are conducted on the benchmark datasets of VQA 2.0 and GQA. The experimental results demonstrate that the proposed method performs favorably against the state-of-the-art approaches. Our best single model has an overall accuracy of 71.18% on the VQA 2.0 dataset and 57.59% on the GQA dataset.

Graph Reasoning Networks for Visual Question Answering

From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

Question-relationship guided graph attention network for visual question answer

Co-attention graph convolutional network for visual question answering

An effective spatial relational reasoning networks for visual question answering

Bilinear Graph Networks for Visual Question Answering

Scene Graph Refinement Network for Visual Question Answering

Joint Learning of Object Graph and Relation Graph for Visual Question Answering

Object-difference Drived Graph Convolutional Networks for Visual Question Answering

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Relation-Aware Graph Attention Network for Visual Question Answering

Graph-enhanced visual representations and question-guided dual attention for visual question answering

Question-Driven Graph Fusion Network For Visual Question Answering

Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering

Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text

Graph-Structured Representations for Visual Question Answering

Hierarchical Semantic Enhanced Directional Graph Network for Visual Commonsense Reasoning

Interpretable Visual Question Answering by Reasoning on Dependency Trees

An Empirical Study on Leveraging Scene Graphs for Visual Question Answering

Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering

Visual Question Answering reasoning with external knowledge based on bimodal graph neural network