Abstract:Existing Visual Question Answering models suffer from the language prior, where the answers provided by the models overly rely on the correlations between questions and answers, ignoring the exact visual information, resulting in a significant drop in the out-of-distribution datasets. To eliminate such language bias, prevalent approaches mainly focus on weakening the language prior with one auxiliary question-only branch while focusing on the statistical question type–answer pairs’ distribution prior rather than that of question–answer pairs. Besides, most models provide the answer with improper visual groundings. This paper proposes a model-agnostic framework to address the above drawbacks by question-conditioned debiasing with focal visual context fusion. To begin with, instead of the question type-conditioned correlations, we overcome the language distribution shortcut from the aspect of question-conditioned correlations by removing the shortcut between questions and the most occurring answer. Additionally, we utilize the deviation of the predicted answer distribution and ground truth as the pseudo target to avoid the model falling into other frequent answers’ distribution bias. Further, we stress the imbalance of the number of images and questions that post higher requirements of a proper visual context. We improve the correct visual utilization ability based on contrastive sampling and design a focal visual context fusion module that incorporates the critical object word extracted from the question after the Part-Of-Speech tagging into the visual features to augment the salient visual information without human annotations. Extensive experiments on the three public benchmark datasets, i.e., VQA v2, VQA-CP v2, and VQA-CP v1, demonstrate the effectiveness of our model.

VQA-PDF: Purifying Debiased Features for Robust Visual Question Answering Task

Debiasing Medical Visual Question Answering via Counterfactual Training

A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering

Debiased Visual Question Answering from Feature and Sample Perspectives.

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Be Flexible! Learn to Debias by Sampling and Prompting for Robust Visual Question Answering.

Debiased Visual Question Answering via the perspective of question types

Digging out Discrimination Information from Generated Samples for Robust Visual Question Answering

Introspective Distillation for Robust Question Answering

Robust visual question answering via polarity enhancement and contrast

Language Bias-Driven Self-Knowledge Distillation with Generalization Uncertainty for Reducing Language Bias in Visual Question Answering

Multi-stage Reasoning on Introspecting and Revising Bias for Visual Question Answering

Robust Visual Question Answering: Datasets, Methods, and Future Challenges

Suppressing Biased Samples for Robust VQA

Question Type-Aware Debiasing for Test-time Visual Question Answering Model Adaptation

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

Vqa-bc: robust visual question answering via bidirectional chaining

Question-conditioned Debiasing with Focal Visual Context Fusion for Visual Question Answering

Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention

Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering