Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

MemexQA: Visual Memex Question Answering

Simple and Effective Visual Question Answering in a Single Modality

Memory Augmented Deep Recurrent Neural Network for Video Question Answering

Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering

Visual Question Answering with Memory-Augmented Networks

Focal Visual-Text Attention for Visual Question Answering

Answering with Memory-Augmented Networks

Motion-Appearance Co-Memory Networks for Video Question Answering

MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Multitask Learning for Visual Question Answering

Long-Term Memory Networks for Question Answering

Video Question Answering Via Multi-Granularity Temporal Attention Network Learning

Video Question Answering Via Gradually Refined Attention over Appearance and Motion

Skip-Thought Memory Networks

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension.

Modular Blended Attention Network for Video Question Answering

Examine Before You Answer: Multi-task Learning with Adaptive-attentions for Multiple-choice VQA.

Multimodal Dual Attention Memory for Video Story Question Answering