Abstract:Most of the recent progresses on visual question answering are based on recurrent neural networks (RNNs) with attention. Despite the success, these models are often timeconsuming and having difficulties in modeling long range dependencies due to the sequential nature of RNNs. We propose a new architecture, Positional Self-Attention with Coattention (PSAC), which does not require RNNs for video question answering. Specifically, inspired by the success of self-attention in machine translation task, we propose a Positional Self-Attention to calculate the response at each position by attending to all positions within the same sequence, and then add representations of absolute positions. Therefore, PSAC can exploit the global dependencies of question and temporal information in the video, and make the process of question and video encoding executed in parallel. Furthermore, in addition to attending to the video features relevant to the given questions (i.e., video attention), we utilize the co-attention mechanism by simultaneously modeling “what words to listen to” (question attention). To the best of our knowledge, this is the first work of replacing RNNs with selfattention for the task of visual question answering. Experimental results of four tasks on the benchmark dataset show that our model significantly outperforms the state-of-the-art on three tasks and attains comparable result on the Count task. Our model requires less computation time and achieves better performance compared with the RNNs-based methods. Additional ablation study demonstrates the effect of each component of our proposed model.

Full-Time Supervision Based Bidirectional RNN for Factoid Question Answering.

An attention mechanism and multi-granularity-based Bi-LSTM model for Chinese Q&A system

LSTM-based Deep Learning Models for Non-Factoid Answer Selection

Bi-directional Long Short-Term Memory Model with Semantic Positional Attention for the Question Answering System

A Question-Focused Multi-Factor Attention Network for Question Answering

Temporality-enhanced Knowledgememory Network for Factoid Question Answering

Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals

Simple Question Answering by Attentive Convolutional Neural Network.

A Neural Network-based Framework for Non-factoid Question Answering

Long-Term Memory Networks for Question Answering

BamnetTL: Bidirectional Attention Memory Network with Transfer Learning for Question Answering Matching

Open Domain Question Answering with Character-Level Deep Learning Models.

Conditioning LSTM Decoder and Bi-directional Attention Based Question Answering System

Question-Aware Memory Network for Multi-hop Question Answering in Human-Robot Interaction

Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs

Temporal Interaction and Causal Influence in Community-Based Question Answering.

Question Answering With Character-Level Lstm Encoders And Model-Based Data Augmentation

Recurrent Memory Reasoning Network for Expert Finding in Community Question Answering

Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering

Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering