Abstract:Visual question answering (VQA) is challenging, because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multimodal feature fusion that is able to capture the complex interactions between multimodal features; and 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a "coattention" mechanism is developed using a deep neural network (DNN) architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multimodal feature fusion, a generalized multimodal factorized high-order pooling approach (MFH) is developed to achieve more effective fusion of multimodal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the Kullback-Leibler divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A DNN architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA data sets and win the runner-up in VQA Challenge 2017.

Resolving Zero-Shot and Fact-Based Visual Question Answering Via Enhanced Fact Retrieval

Simple and Effective Visual Question Answering in a Single Modality

Zero-Shot Visual Question Answering Using Knowledge Graph

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Knowledge-aware image understanding with multi-level visual representation enhancement for visual question answering

FVQA: Fact-Based Visual Question Answering

Question guided multimodal receptive field reasoning network for fact-based visual question answering

A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA.

Boosting Visual Question Answering with Context-aware Knowledge Aggregation

Fact-based Visual Question Answering Via Dual-Process System

Visual Question Answering As Reading Comprehension

Visual Question Answering by Pattern Matching and Reasoning

An effective spatial relational reasoning networks for visual question answering

Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion

Good Questions Help Zero-Shot Image Reasoning

Knowledge Condensation and Reasoning for Knowledge-based VQA

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

Visual Question Answering Via Combining Inferential Attention and Semantic Space Mapping

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering.