Abstract:Though beneficial for encouraging the visual question answering (VQA) models to discover the underlying knowledge by exploiting the input-output correlation beyond image and text contexts, the existing knowledge VQA data sets are mostly annotated in a crowdsource way, e.g., collecting questions and external reasons from different users via the Internet. In addition to the challenge of knowledge reasoning, how to deal with the annotator bias also remains unsolved, which often leads to superficial overfitted correlations between questions and answers. To address this issue, we propose a novel data set named knowledge-routed visual question reasoning for VQA model evaluation. Considering that a desirable VQA model should correctly perceive the image context, understand the question, and incorporate its learned knowledge, our proposed data set aims to cut off the shortcut learning exploited by the current deep embedding models and push the research boundary of the knowledge-based visual question reasoning. Specifically, we generate the question-answer pair based on both the visual genome scene graph and an external knowledge base with controlled programs to disentangle the knowledge from other biases. The programs can select one or two triplets from the scene graph or knowledge base to push multistep reasoning, avoid answer ambiguity, and balance the answer distribution. In contrast to the existing VQA data sets, we further imply the following two major constraints on the programs to incorporate knowledge reasoning. First, multiple knowledge triplets can be related to the question, but only one knowledge relates to the image object. This can enforce the VQA model to correctly perceive the image instead of guessing the knowledge based on the given question solely. Second, all questions are based on different knowledge, but the candidate answers are the same for both the training and test sets. We make the testing knowledge unused during training to evaluate whether a model can understand question words and handle unseen combinations. Extensive experiments with various baselines and state-of-the-art VQA models are conducted to demonstrate that there still exists a big gap between the model with and without groundtruth supporting triplets when given the embedded knowledge base. This reveals the weakness of the current deep embedding models on the knowledge reasoning problem.

A Knowledge-Enhanced Inferential Network for Cross-Modality Multi-hop VQA

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network

Knowledge-Enhanced Visual Question Answering with Multi-modal Joint Guidance.

Perceptual Visual Reasoning with Knowledge Propagation

Question guided multimodal receptive field reasoning network for fact-based visual question answering

Knowledge-Based Visual Question Answering Using Multi-Modal Semantic Graph

Visual Question Answering reasoning with external knowledge based on bimodal graph neural network

Multi-source Multi-level Attention Networks for Visual Question Answering

Multi-scale Relation Reasoning for Multi-Modal Visual Question Answering.

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Knowledge-Routed Visual Question Reasoning: Challenges for Deep Representation Embedding

Knowledge-aware image understanding with multi-level visual representation enhancement for visual question answering

Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering

II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering

Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering

Open-Set Knowledge-Based Visual Question Answering with Inference Paths

Graph Reasoning Networks for Visual Question Answering

Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text

DSAMR: Dual-Stream Attention Multi-hop Reasoning for knowledge-based visual question answering

Hierarchical Multi-Modality Graph Reasoning for Remote Sensing Visual Question Answering