Abstract:Existing methods for video question answering (VideoQA) often suffer from spurious correlations between different modalities, leading to a failure in identifying the dominant visual evidence and the intended question. Moreover, these methods function as black boxes, making it difficult to interpret the visual scene during the QA process. In this paper, to discover critical video segments and frames that serve as the visual causal scene for generating reliable answers, we present a causal analysis of VideoQA and propose a framework for cross-modal causal relational reasoning, named Visual Causal Scene Refinement (VCSR). Particularly, a set of causal front-door intervention operations is introduced to explicitly find the visual causal scenes at both segment and frame levels. Our VCSR involves two essential modules: i) the Question-Guided Refiner (QGR) module, which refines consecutive video frames guided by the question semantics to obtain more representative segment features for causal front-door intervention; ii) the Causal Scene Separator (CSS) module, which discovers a collection of visual causal and non-causal scenes based on the visual-linguistic causal relevance and estimates the causal effect of the scene-separating intervention in a contrastive learning manner. Extensive experiments on the NExT-QA, Causal-VidQA, and MSRVTT-QA datasets demonstrate the superiority of our VCSR in discovering visual causal scene and achieving robust video question answering. The code is available at <a class="link-external link-https" href="https://github.com/YangLiu9208/VCSR" rel="external noopener nofollow">this https URL</a>.

Reducing Vision-Answer Biases for Multiple-Choice VQA

Simple and Effective Visual Question Answering in a Single Modality

Dual Path Multi-Modal High-Order Features for Textual Content Based Visual Question Answering

Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA

Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning

Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering

Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention

Learning content and context with language bias for Visual Question Answering

Visual Question Answering Method Based on Counterfactual Thinking

Cross-Modal Contrastive Learning for Robust Reasoning in VQA

Multitask Learning for Visual Question Answering

Counterfactual Causal-Effect Intervention for Interpretable Medical Visual Question Answering

Vqa-bc: robust visual question answering via bidirectional chaining

Removing Bias of Video Question Answering by Causal Theory

Visual Causal Scene Refinement for Video Question Answering

Causal Reasoning through Two Layers of Cognition for Improving Generalization in Visual Question Answering

Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention

Debiased Visual Question Answering via the perspective of question types

Multi-Question Learning for Visual Question Answering

SC-ML: Self-supervised Counterfactual Metric Learning for Debiased Visual Question Answering