Abstract:Visual question answering (VQA) is a prevalent task in real-world, and plays an essential role in helping the blind understand the physical world. However, due to the real-world complexity, VQA test samples may come from a different distribution from the training data, resulting in unavoidable performance degradation. This similar issue also exists in the image recognition field, in which one most recent effective solutions is a test-time adaptation (TTA). TTA adapts a trained model at test time using only test samples, which provides a new idea to alleviate the analogous issue in VQA. However, naively introducing existing TTA methods (e.g., test-time entropy minimisation) into VQA is imperfect and achieves only marginal performance gain. The reason is that prior methods do not consider the special nature of the VQA problem and ignore that 1) the biased samples in the dataset may have negative effects on test-time model adaptation, and 2) the model may have captured the biases in the dataset. In this paper, we propose Test-time Debiased Self-supervised (TDS) learning objectives for VQA model adaptation. Specifically, we minimise the entropy for those unbiased test samples. To identify these samples, we construct a negative sample for each test sample, and regard the test samples as unbiased if the output answers are different when feeding the test sample and the counterpart negative sample into the VQA model. Meanwhile, we also remove those samples with high prediction entropy from adaptation, making the test-time gradients more reliable. To hinder the model from excessively fitting the superficial correlations of the biased sample, we adopt the biased samples and the counterpart negative samples to assist the adaptation. Extensive experiments on the VQA-CP v1 and VQA-CP v2 datasets demonstrate the effectiveness of our TDS.

Question Type-Aware Debiasing for Test-time Visual Question Answering Model Adaptation

Test-Time Model Adaptation for Visual Question Answering with Debiased Self-Supervisions

Debiased Visual Question Answering via the perspective of question types

Debiasing Medical Visual Question Answering via Counterfactual Training

Be Flexible! Learn to Debias by Sampling and Prompting for Robust Visual Question Answering.

SQT: Debiased Visual Question Answering via Shuffling Question Types

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Debiased Visual Question Answering from Feature and Sample Perspectives.

A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering

Question-conditioned Debiasing with Focal Visual Context Fusion for Visual Question Answering

VQA-PDF: Purifying Debiased Features for Robust Visual Question Answering Task

Suppressing Biased Samples for Robust VQA

Efficient Counterfactual Debiasing for Visual Question Answering

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasets

Adaptive Re-Balancing Network with Gate Mechanism for Long-Tailed Visual Question Answering.

Digging out Discrimination Information from Generated Samples for Robust Visual Question Answering

Alleviating Shortcut Learning Behavior of VQA Model with Context Augmentation and Adaptive Loss Adjustment

Language Bias-Driven Self-Knowledge Distillation with Generalization Uncertainty for Reducing Language Bias in Visual Question Answering

A Robust Visual Question Answering Approach to Reduce Multimodal Bias

Greedy Gradient Ensemble for Robust Visual Question Answering