Abstract:Today's VQA models still tend to capture superficial linguistic correlations in the training set and fail to generalize to the test set with different QA distributions. To reduce these language biases, recent VQA works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on diagnostic benchmarks for out-of-distribution testing. However, due to the complex model design, ensemble-based methods are unable to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable: The model should rely on the right visual regions when making decisions. 2) Question-sensitive: The model should be sensitive to the linguistic variations in questions. To this end, we propose a novel model-agnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. After training with CSST, VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST). CSS generates counterfactual samples by carefully masking critical objects in images or words in questions and assigning pseudo ground-truth answers. CST not only trains the VQA models with both complementary samples to predict respective ground-truth answers, but also urges the VQA models to further distinguish the original samples and superficially similar counterfactual ones. To facilitate the CST training, we propose two variants of supervised contrastive loss for VQA, and design an effective positive and negative sample selection mechanism based on CSS. Extensive experiments have shown the effectiveness of CSST. Particularly, by building on top of model LMH+SAR [1], [2], we achieve record-breaking performance on all out-of-distribution benchmarks (e.g., VQA-CP v2, VQA-CP v1, and GQA-OOD).

Causal Inference with Knowledge Distilling and Curriculum Learning for Unbiased VQA

Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA

Knowledge Condensation and Reasoning for Knowledge-based VQA

Causal Reasoning through Two Layers of Cognition for Improving Generalization in Visual Question Answering

Removing Bias of Video Question Answering by Causal Theory

From Superficial to Deep: Language Bias Driven Curriculum Learning for Visual Question Answering.

Reducing Vision-Answer Biases for Multiple-Choice VQA

Zero-Shot Visual Question Answering Using Knowledge Graph

Inverse Visual Question Answering: A New Benchmark and VQA Diagnosis Tool

Knowledge-Enhanced Visual Question Answering with Multi-modal Joint Guidance.

Vqa-bc: robust visual question answering via bidirectional chaining

Knowledge-Based Counterfactual Queries for Visual Question Answering

Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering

Visual Causal Scene Refinement for Video Question Answering

Visual Grounding Methods for VQA are Working for the Wrong Reasons!

Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA

Counterfactual Causal-Effect Intervention for Interpretable Medical Visual Question Answering

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering

Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering

MHKD-MVQA: Multimodal Hierarchical Knowledge Distillation for Medical Visual Question Answering.