Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

Triple-Joint Modeling for Question Generation Using Cross-Task Autoencoder.

Joint Learning of Question Answering and Question Generation

Question Answering and Question Generation as Dual Tasks

Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

GTM: A Generative Triple-Wise Model for Conversational Question Generation

Consecutive Question Generation via Dynamic Multitask Learning

Enhancing Question Generation with Commonsense Knowledge

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

Learning to Generate Question by Asking Question: A Primal-Dual Approach with Uncommon Word Generation

Multi-Task Learning with Language Modeling for Question Generation

Dual Learning for Visual Question Generation.

Large-Scale Question Tagging Via Joint Question-Topic Embedding Learning.

Video Question Answering Via Grounded Cross-Attention Network Learning.

Jointly Learning Attentions With Semantic Cross-Modal Correlation For Visual Question Answering

Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner

Query2Triple: Unified Query Encoding for Answering Diverse Complex Queries over Knowledge Graphs

Unified Question Generation with Continual Lifelong Learning

Learning To Ask More: Semi-Autoregressive Sequential Question Generation Under Dual-Graph Interaction

Question Answering With Character-Level Lstm Encoders And Model-Based Data Augmentation

Context-aware Multi-level Question Embedding Fusion for visual question answering