Abstract:The objective of visual question answering (VQA) is to adequately comprehend a question and identify relevant contents in an image that can provide an answer. Existing approaches in VQA often combine visual and question features directly to create a unified cross-modality representation for answer inference. However, this kind of approach fails to bridge the semantic gap between visual and text modalities, resulting in a lack of alignment in cross-modality semantics and the inability to match key visual content accurately. In this article, we propose a model called the caption bridge-based cross-modality alignment and contrastive learning model (CBAC) to address the issue. The CBAC model aims to reduce the semantic gap between different modalities. It consists of a caption-based cross-modality alignment module and a visual-caption (V-C) contrastive learning module. By utilizing an auxiliary caption that shares the same modality as the question and has closer semantic associations with the visual, we are able to effectively reduce the semantic gap by separately matching the caption with both the question and the visual to generate pre-alignment features for each, which are then used in the subsequent fusion process. We also leverage the fact that V-C pairs exhibit stronger semantic connections compared to question-visual (Q-V) pairs to employ a contrastive learning mechanism on visual and caption pairs to further enhance the semantic alignment capabilities of single-modality encoders. Extensive experiments conducted on three benchmark datasets demonstrate that the proposed model outperforms previous state-of-the-art VQA models. Additionally, ablation experiments confirm the effectiveness of each module in our model. Furthermore, we conduct a qualitative analysis by visualizing the attention matrices to assess the reasoning reliability of the proposed model.

QAVidCap: Enhancing Video Captioning Through Question Answering Techniques

Learning to enhance areal video captioning with visual question answering

All You May Need for VQA are Image Captions

Video Interactive Captioning with Human Prompts.

Image Captioning by Asking Questions

CapWAP: Captioning with a Purpose

The nature of respiratory changes associated with sleep onset.

Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Bridging the Cross-Modality Semantic Gap in Visual Question Answering

End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Sampling

Learning Video-Text Aligned Representations for Video Captioning

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Frame Augmented Alternating Attention Network for Video Question Answering.

Zero-shot Visual Question Answering with Language Model Feedback

Enhancing machine vision: the impact of a novel innovative technology on video question-answering

Question-controlled Text-aware Image Captioning

Video Question Answering: a Survey of Models and Datasets

Delving Deeper into the Decoder for Video Captioning

Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner

Question-Aware Global-Local Video Understanding Network for Audio-Visual Question Answering