Abstract:As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations (i.e., moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel M ulti- M odal C ross- D omain A lignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; and (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Partial Annotation-based Video Moment Retrieval Via Iterative Learning

Video Moment Retrieval with Noisy Labels

Hybrid-Learning Video Moment Retrieval across Multi-Domain Labels

Faster Video Moment Retrieval with Point-Level Supervision

Semantic Video Moment Retrieval by Temporal Feature Perturbation and Refinement

Are Binary Annotations Sufficient? Video Moment Retrieval Via Hierarchical Uncertainty-Based Active Learning

VLANet: Video-Language Alignment Network for Weakly-Supervised Video Moment Retrieval

Fine-Grained Modality Relation-Aware Network for Video Moment Retrieval

Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval Using Language

Video Moment Retrieval from Text Queries via Single Frame Annotation

MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval

Weakly-Supervised Video Moment Retrieval Via Semantic Completion Network

Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training

Context-Enhanced Video Moment Retrieval with Large Language Models

Zero-Shot Video Moment Retrieval with Angular Reconstructive Text Embeddings

Siamese Alignment Network for Weakly Supervised Video Moment Retrieval

Generative Video Diffusion for Unseen Cross-Domain Video Moment Retrieval

Query As Supervision: Towards Low-Cost and Robust Video Moment and Highlight Retrieval

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

QD-VMR: Query Debiasing with Contextual Understanding Enhancement for Video Moment Retrieval

Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models