Abstract:In recent years, Visual Question Answering (VQA) has attracted increasing attention due to its requirement on cross-modal understanding and reasoning of vision and language. VQA is proposed to automatically answer natural language questions with reference to a given image. VQA is challenging, because the reasoning process on a visual domain needs a full understanding of the spatial relationship, semantic concepts, as well as the common sense for a real image. However, most existing approaches jointly embed the abstract low-level visual features and high-level question features to infer answers. These works have limited reasoning ability due to the lack of modeling of the rich spatial context of regions, high-level semantics of images, and knowledge across multiple sources. To solve the challenges, we propose multi-source multi-level attention networks for visual question answering that can benefit both spatial inferences by visual attention on context-aware region representation and reasoning by semantic attention on concepts as well as external knowledge. Indeed, we learn to reason on image representation by question-guided attention at different levels across multiple sources, including region and concept level representation from image source as well as sentence level representation from the external knowledge base. First, we encode region-based middle-level outputs from Convolutional Neural Networks (CNNs) into spatially embedded representation by a multi-directional two-dimensional recurrent neural network and, further, locate the answer-related regions by Multiple Layer Perceptron as visual attention. Second, we generate semantic concepts from high-level semantics in CNNs and select those question-related concepts as concept attention. Third, we query semantic knowledge from the general knowledge base by concepts and selected question-related knowledge as knowledge attention. Finally, we jointly optimize visual attention, concept attention, knowledge attention, and question embedding by a softmax classifier to infer the final answer. Extensive experiments show the proposed approach achieved significant improvement on two very challenging VQA datasets.

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Video Question Answering Via Grounded Cross-Attention Network Learning.

Video Question Answering via Attribute-Augmented Attention Network Learning

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Video Question Answering Via Hierarchical Dual-Level Attention Network Learning.

Multi-Turn Video Question Answering Via Multi-Stream Hierarchical Attention Context Network

Frame Augmented Alternating Attention Network for Video Question Answering.

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

Graph-Based Multi-Interaction Network for Video Question Answering

Multi-Turn Video Question Answering via Hierarchical Attention Context Reinforced Networks

Unifying the Video and Question Attentions for Open-Ended Video Question Answering.

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Hierarchical Recurrent Contextual Attention Network for Video Question Answering

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Video Question Answering Via Hierarchical Spatio-Temporal Attention Networks

TASTA: Text-Assisted Spatial and Temporal Attention Network for Video Question Answering.

Multi-Turn Video Question Generation Via Reinforced Multi-Choice Attention Network

Multi-source Multi-level Attention Networks for Visual Question Answering