Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network

Multi-Granularity Interaction and Integration Network for Video Question Answering

Frame Augmented Alternating Attention Network for Video Question Answering.

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Modular Blended Attention Network for Video Question Answering

Video Question Answering Via Grounded Cross-Attention Network Learning.

Graph-Based Multi-Interaction Network for Video Question Answering

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Video Question Answering with Semantic Disentanglement and Reasoning

Multi-interaction Network with Object Relation for Video Question Answering

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering.

QAVidCap: Enhancing Video Captioning Through Question Answering Techniques

Multichannel Attention Refinement for Video Question Answering.

Cross-Attentional Spatio-Temporal Semantic Graph Networks for Video Question Answering

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Frame augmented alternating attention network for video question answering

Structured Two-stream Attention Network for Video Question Answering

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering