Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering

Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering

Multi-Question Learning for Visual Question Answering

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Modality attention fusion model with hybrid multi-head self-attention for video understanding

Long-Term Video Question Answering Via Multimodal Hierarchical Memory Attentive Networks

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

TASTA: Text-Assisted Spatial and Temporal Attention Network for Video Question Answering.

Complementary Spatiotemporal Network for Video Question Answering.

Triple Multimodal Cyclic Fusion and Self-Adaptive Balancing for Video Q&A Systems

Hierarchical Temporal Fusion of Multi-grained Attention Features for Video Question Answering

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Fusing Temporally Distributed Multi-Modal Semantic Clues for Video Question Answering.

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

Multichannel Attention Refinement for Video Question Answering.

Video Question Answering with Semantic Disentanglement and Reasoning

Advancing Video Question Answering with a Multi-modal and Multi-layer Question Enhancement Network

MT-VQA: A Multi-task Approach for Quality Assessment of Short-form Videos