Abstract:Long-term Video Question Answering (VideoQA) is a challenging vision-and-language bridging task focusing on semantic understanding of untrimmed long-term videos and diverse free-form questions, simultaneously emphasizing comprehensive cross-modal reasoning to yield precise answers. The canonical approaches often rely on off-the-shelf feature extractors to detour the expensive computation overhead, but often result in domain-independent modality-unrelated representations. Furthermore, the inherent gradient blocking between unimodal comprehension and cross-modal interaction hinders reliable answer generation. In contrast, recent emerging successful video-language pre-training models enable cost-effective end-to-end modeling but fall short in domain-specific ratiocination and exhibit disparities in task formulation. Toward this end, we present an entirely end-to-end solution for long-term VideoQA: Multi-granularity Contrastive cross-modal collaborative Generation (MCG) model. To derive discriminative representations possessing high visual concepts, we introduce Joint Unimodal Modeling (JUM) on a clip-bone architecture and leverage Multi-granularity Contrastive Learning (MCL) to harness the intrinsically or explicitly exhibited semantic correspondences. To alleviate the task formulation discrepancy problem, we propose a Cross-modal Collaborative Generation (CCG) module to reformulate VideoQA as a generative task instead of the conventional classification scheme, empowering the model with the capability for cross-modal high-semantic fusion and generation so as to rationalize and answer. Extensive experiments conducted on six publicly available VideoQA datasets underscore the superiority of our proposed method.

CREAM: Coarse-to-Fine Retrieval and Multi-modal Efficient Tuning for Document VQA

Dual Path Multi-Modal High-Order Features for Textual Content Based Visual Question Answering

Simple and Effective Visual Question Answering in a Single Modality

Multi-Page Document Visual Question Answering using Self-Attention Scoring Mechanism

CAMVR: Context-Adaptive Multi-View Representation Learning for Dense Retrieval

PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answering

Two-Stage Multimodality Fusion for High-Performance Text-Based Visual Question Answering.

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

Spontaneous regression of orbital Langerhans cell granulomatosis in a three-year-old girl.

Multiscale Feature Extraction and Fusion of Image and Text in VQA

Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion

Multi-Modal Learning with Text Merging for TEXTVQA.

Multitask Learning for Visual Question Answering

Multi-Question Learning for Visual Question Answering

MGA-VQA: Multi-Granularity Alignment for Visual Question Answering

QD-VMR: Query Debiasing with Contextual Understanding Enhancement for Video Moment Retrieval

Context-aware Multi-level Question Embedding Fusion for visual question answering

Beyond OCR + VQA: Towards End-to-End Reading and Reasoning for Robust and Accurate TextVQA

Multi-view Content-aware Indexing for Long Document Retrieval