Abstract:The objective of visual question answering (VQA) is to adequately comprehend a question and identify relevant contents in an image that can provide an answer. Existing approaches in VQA often combine visual and question features directly to create a unified cross-modality representation for answer inference. However, this kind of approach fails to bridge the semantic gap between visual and text modalities, resulting in a lack of alignment in cross-modality semantics and the inability to match key visual content accurately. In this article, we propose a model called the caption bridge-based cross-modality alignment and contrastive learning model (CBAC) to address the issue. The CBAC model aims to reduce the semantic gap between different modalities. It consists of a caption-based cross-modality alignment module and a visual-caption (V-C) contrastive learning module. By utilizing an auxiliary caption that shares the same modality as the question and has closer semantic associations with the visual, we are able to effectively reduce the semantic gap by separately matching the caption with both the question and the visual to generate pre-alignment features for each, which are then used in the subsequent fusion process. We also leverage the fact that V-C pairs exhibit stronger semantic connections compared to question-visual (Q-V) pairs to employ a contrastive learning mechanism on visual and caption pairs to further enhance the semantic alignment capabilities of single-modality encoders. Extensive experiments conducted on three benchmark datasets demonstrate that the proposed model outperforms previous state-of-the-art VQA models. Additionally, ablation experiments confirm the effectiveness of each module in our model. Furthermore, we conduct a qualitative analysis by visualizing the attention matrices to assess the reasoning reliability of the proposed model.

Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning

Human-like Controllable Image Captioning with Verb-specific Semantic Roles

Improving Image Captioning with Better Use of Caption

Bridging the Cross-Modality Semantic Gap in Visual Question Answering

Exploring Visual Relationship for Image Captioning

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering

Exploring Explicit and Implicit Visual Relationships for Image Captioning

Bridge the Gap: High-level Semantic Planning for Image Captioning

Image Captioning with Scene-graph Based Semantic Concepts.

Adaptively Attending to Visual Attributes and Linguistic Knowledge for Captioning

Improving Image Captioning with Better Use of Captions

Bridge Video and Text with Cascade Syntactic Structure.

Bridging The Domain Gap Arising from Text Description Differences for Stable Text-To-Image Generation.

Controllable image caption with an encoder-decoder optimization structure

Towards Bridging Video and Language by Caption Generation and Sentence Localization.

Bounding and Filling: A Fast and Flexible Framework for Image Captioning

Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

From Captions to Visual Concepts and Back

Top-down framework for weakly-supervised grounded image captioning

Neural Image Caption Generation with Weighted Training and Reference

Bi-Directional Co-Attention Network for Image Captioning