Counterfactual Visual Dialog: Robust Commonsense Knowledge Learning from Unbiased Training
An-An Liu,Chenxi Huang,Ning Xu,Hongshuo Tian,Jing Liu,Yongdong Zhang
DOI: https://doi.org/10.1109/tmm.2023.3284594
IF: 7.3
2023-01-01
IEEE Transactions on Multimedia
Abstract:Visual Dialog (VD) requires an agent to answer the current question by engaging in a conversation with humans referring to an image. Despite the recent progress, it is beneficial to introduce external commonsense knowledge to fully understand the given image and dialog history. However, the existing knowledge-based VD models are inclined to rely on severe learning bias brought by commonsense, e.g., the retrieved $\lt {\mathtt{bus}}, {\mathtt{capable\;of}}, {\mathtt{transport\;people}}\gt $, $\lt {\mathtt{bus}}, {\mathtt{is\;a}}, {\mathtt{public\;transport}}\gt $, and $\lt {\mathtt{bus}}, {\mathtt{is\;a}}, {\mathtt{car}}\gt $ can induce a spurious correlation between the question “What is the bus used for?” and the false answer “City bus”. There are two challenges to make commonsense learning more robust against spurious correlations: 1) how to disentangle the true effect of “good” commonsense knowledge from the whole, and 2) how to estimate and remove the effect of “bad” commonsense bias on answers. In this paper, we propose a novel CounterFactual Commonsense learning scheme for the Visual Dialog task (CFC-VD). First, comparing with the causal graph of existing VD models, we add one new commonsense node and one new link to multi-modal information from history, question, and image. Since the retrieved knowledge prior is subtle and uncontrollable, we consider it as an unobserved confounder in the commonsense node, which leads to spurious correlations for the answer inference. Then, to remove the effect of the confounder, we formulate it as the direct causal effect of commonsense on answers and remove the direct language effect by subtracting it from the total causal effect via counterfactual reasoning. Experimental results certify the effectiveness of our method on the prevailing Visdial v0.9 and Visdial v1.0 datasets.
computer science, information systems,telecommunications, software engineering