Abstract:AbstractVisual Question Answering (VQA) is a challenging task that has gained increasing attention from both the computer vision and the natural language processing communities in recent years. Given a question in natural language, a VQA system is designed to automatically generate the answer according to the referenced visual content. Though there recently has been much intereset in this topic, the existing work of visual question answering mainly focuses on a single static image, which is only a small part of the dynamic and sequential visual data in the real world. As a natural extension, video question answering (VideoQA) is less explored. Because of the inherent temporal structure in the video, the approaches of ImageQA may be ineffectively applied to video question answering. In this article, we not only take the spatial and temporal dimension of video content into account but also employ an external knowledge base to improve the answering ability of the network. More specifically, we propose a knowledge-based progressive spatial-temporal attention network to tackle this problem. We obtain both objects and region features of the video frames from a region proposal network. The knowledge representation is generated by a word-level attention mechanism using the comment information of each object that is extracted from DBpedia. Then, we develop a question-knowledge-guided progressive spatial-temporal attention network to learn the joint video representation for video question answering task. We construct a large-scale video question answering dataset. The extensive experiments based on two different datasets validate the effectiveness of our method.

Initialized Frame Attention Networks for Video Question Answering.

Frame Augmented Alternating Attention Network for Video Question Answering.

Frame augmented alternating attention network for video question answering

Video question answering by frame attention

Video Question Answering Via Grounded Cross-Attention Network Learning.

Video Question Answering via Attribute-Augmented Attention Network Learning

TASTA: Text-Assisted Spatial and Temporal Attention Network for Video Question Answering.

Spatio-Temporal Context Networks for Video Question Answering.

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Structured Two-stream Attention Network for Video Question Answering

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Compositional Attention Networks with Two-Stream Fusion for Video Question Answering

Spatiotemporal-Textual Co-Attention Network for Video Question Answering

Feature Augmented Memory with Global Attention Network for VideoQA

Memory Augmented Deep Recurrent Neural Network for Video Question Answering

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Relation-aware Hierarchical Attention Framework for Video Question Answering