Abstract:Medical Visual Question Answering (Med-VQA) is expected to predict a convincing answer with the given medical image and clinical question, aiming to assist clinical decision-making. While today's works have intention to rely on the superficial linguistic correlations as a shortcut, which may generate emergent dissatisfactory clinic answers. In this paper, we propose a novel DeBiasing Med-VQA model with CounterFactual training (DeBCF) to overcome language priors comprehensively. Specifically, we generate counterfactual samples by masking crucial keywords and assigning irrelevant labels, which implicitly promotes the sensitivity of the model to the semantic words and visual objects for bias-weaken. Furthermore, to explicitly prevent the cheating linguistic correlations, we formulate the language prior into counterfactual causal effects and eliminate it from the total effect on the generated answers. Additionally, we initiatively present a newly splitting bias-sensitive Med-VQA dataset, Semantically-Labeled Knowledge-Enhanced under Changing Priors (SLAKE-CP) dataset through regrouping and re-splitting the train-set and test-set of SLAKE into the different prior distribution of answers, dedicating the model to learn interpretable objects rather than overwhelmingly memorizing biases. Experimental results on two public datasets and SLAKE-CP demonstrate that the proposed DeBCF outperforms existing state-of-the-art Med-VQA models and obtains significant improvement in terms of accuracy and interpretability. To our knowledge, it's the first attempt to overcome language priors in Med-VQA and construct the bias-sensitive dataset for evaluating debiased ability.

CausalME: Balancing bi-modalities in Visual Question Answering.

Dual Path Multi-Modal High-Order Features for Textual Content Based Visual Question Answering

Modality Re-Balance for Visual Question Answering: A Causal Framework.

Simple and Effective Visual Question Answering in a Single Modality

Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

Reducing Vision-Answer Biases for Multiple-Choice VQA

Debiasing Medical Visual Question Answering via Counterfactual Training

Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering

Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

Removing Bias of Video Question Answering by Causal Theory

An Empirical Study on the Language Modal in Visual Question Answering

From Superficial to Deep: Language Bias Driven Curriculum Learning for Visual Question Answering.

Discovering the Real Association: Multimodal Causal Reasoning in Video Question Answering

Visual Causal Scene Refinement for Video Question Answering

Counterfactual VQA: A Cause-Effect Look at Language Bias.

A Robust Dual-debiasing VQA Model Based on Counterfactual Causal Effect

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering