Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

Modular Blended Attention Network for Video Question Answering

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Cross-Modal Multistep Fusion Network with Co-Attention for Visual Question Answering

Multi-interaction Network with Object Relation for Video Question Answering

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Adversarial Multimodal Network for Movie Question Answering

MCQA: Multimodal Co-attention Based Network for Question Answering

Frame Augmented Alternating Attention Network for Video Question Answering.

Graph-Based Multi-Interaction Network for Video Question Answering

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

Modality attention fusion model with hybrid multi-head self-attention for video understanding

Hierarchical Attention Networks for Multimodal Machine Learning

Multimodal Cross-guided Attention Networks for Visual Question Answering

Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering

Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering.

Dynamic Spatio-Temporal Modular Network for Video Question Answering

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Multi-Channel Co-Attention Network For Visual Question Answering

Multichannel Attention Refinement for Video Question Answering.

Multi-Granularity Interaction and Integration Network for Video Question Answering