Abstract:The problem of visual question answering (VQA) is of significant importance both as a challenging research question and for the rich set of applications it enables. In this context, however, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in VQA models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of VQA and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset (Antol et al., in: ICCV, 2015) by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at http://visualqa.org/ as part of the 2nd iteration of the VQA Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. We also present interesting insights from analysis of the participant entries in VQA Challenge 2017, organized by us on the proposed VQA v2.0 dataset. The results of the challenge were announced in the 2nd VQA Challenge Workshop at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

Beyond Words: ESC‐Net Revolutionizes VQA by Elevating Visual Features and Defying Language Priors

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Simple and Effective Visual Question Answering in a Single Modality

Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder

Overcoming Language Priors in Visual Question Answering with Adversarial Regularization

The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Visual Grounding Methods for VQA are Working for the Wrong Reasons!

Overcoming language priors with self-contrastive learning for visual question answering

An effective spatial relational reasoning networks for visual question answering

Enhancing scene‐text visual question answering with relational reasoning, attention and dynamic vocabulary integration

Modulated Self-attention Convolutional Network for VQA

Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy

Visual Question Answering in Remote Sensing with Cross-Attention and Multimodal Information Bottleneck

VSCA: A Sentence Matching Model Incorporating Visual Perception

VisQA: X-raying Vision and Language Reasoning in Transformers

An Empirical Study on the Language Modal in Visual Question Answering

Question-Guided Semantic Dual-Graph Visual Reasoning with Novel Answers.

From Pixels to Objects: Cubic Visual Attention for Visual Question Answering

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering