Abstract:As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations (i.e., moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel M ulti- M odal C ross- D omain A lignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; and (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Adversarial Multi-Grained Embedding Network for Cross-Modal Text-Video Retrieval

Fine-grained Cross-modal Alignment Network for Text-Video Retrieval

Visual Spatio-temporal Relation-enhanced Network for Cross-modal Text-Video Retrieval

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Cross-modal Embeddings for Video and Audio Retrieval

Annotation Efficient Cross-Modal Retrieval with Adversarial Attentive Alignment

Coarse-to-fine dual-level attention for video-text cross modal retrieval

Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval

Cross-modal Search Method of Technology Video based on Adversarial Learning and Feature Fusion

Adversarial Video Moment Retrieval by Jointly Modeling Ranking and Localization

Adversarial Cross-Modal Retrieval

Joint embeddings with multimodal cues for video-text retrieval

Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network

Adversarial Multimodal Network for Movie Question Answering

Multi-level Alignment Network for Domain Adaptive Cross-modal Retrieval.

Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representations

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

BiC-Net: Learning Efficient Spatio-Temporal Relation for Text-Video Retrieval