Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Video question answering via gradually refined attention over appearance and motion

Multichannel Attention Refinement for Video Question Answering.

Video question answering by frame attention

Unifying the Video and Question Attentions for Open-Ended Video Question Answering.

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Frame Augmented Alternating Attention Network for Video Question Answering.

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Video Question Answering Via Grounded Cross-Attention Network Learning.

TASTA: Text-Assisted Spatial and Temporal Attention Network for Video Question Answering.

Video Question Answering via Attribute-Augmented Attention Network Learning

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Sampling

Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering.

QAVidCap: Enhancing Video Captioning Through Question Answering Techniques

Hierarchical Recurrent Contextual Attention Network for Video Question Answering

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Video Question Answering with Semantic Disentanglement and Reasoning