Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning

Feilong Chen,Xiuyi Chen,Shuang Xu,Bo Xu

DOI: https://doi.org/10.48550/arXiv.2204.07302

2022-04-15

Abstract:Visual Dialog is a challenging vision-language task since the visual dialog agent needs to answer a series of questions after reasoning over both the image content and dialog history. Though existing methods try to deal with the cross-modal understanding in visual dialog, they are still not enough in ranking candidate answers based on their understanding of visual and textual contexts. In this paper, we analyze the cross-modal understanding in visual dialog based on the vision-language pre-training model VD-BERT and propose a novel approach to improve the cross-modal understanding for visual dialog, named ICMU. ICMU enhances cross-modal understanding by distinguishing different pulled inputs (i.e. pulled images, questions or answers) based on four-way contrastive learning. In addition, ICMU exploits the single-turn visual question answering to enhance the visual dialog model's cross-modal understanding to handle a multi-turn visually-grounded conversation. Experiments show that the proposed approach improves the visual dialog model's cross-modal understanding and brings satisfactory gain to the VisDial dataset.

Computer Vision and Pattern Recognition,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve cross - modal understanding ability in the Visual Dialog task. Specifically, when dealing with visual dialog, although existing methods have tried various ways to understand the relationship between image content and dialog history, there are still deficiencies in ranking candidate answers based on the understanding of visual and textual context. This causes the model to often rate irrelevant wrong answers too highly, affecting the quality and accuracy of the dialog.

To improve this situation, based on the pre - trained model VD - BERT, the author proposes a new method - ICMU (Improving Cross - Modal Understanding). ICMU enhances the discrimination ability between different inputs (such as images, questions or answers) through four - way contrastive learning, and uses single - turn visual question answering (VQA) to further improve the cross - modal understanding ability of the model, so as to better handle multi - turn visual - based dialogs.

Through experimental verification, the ICMU method not only significantly improves the cross - modal understanding ability of the model on the VisDial dataset, but also achieves satisfactory results on multiple evaluation metrics, proving its effectiveness and practicality.