Abstract:As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations (i.e., moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel M ulti- M odal C ross- D omain A lignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; and (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Cross Modal Domain Generalization for Query-based Video Segmentation

Domain Adaptation Meets Zero-Shot Learning: an Annotation-Efficient Approach to Multi-Modality Medical Image Segmentation

BEV-DG: Cross-Modal Learning under Bird's-Eye View for Domain Generalization of 3D Semantic Segmentation

VideoDG: Generalizing Temporal Relations in Videos to Novel Domains

Exploring Multi-Modality Structure for Cross Domain Adaptation in Video Concept Annotation

Semantic-conditioned Dual Adaptation for Cross-domain Query-based Visual Segmentation

Multi-Modal Video Topic Segmentation with Dual-Contrastive Domain Adaptation

Adversarial Consistency for Single Domain Generalization in Medical Image Segmentation

Cross-modality Online Distillation for Multi-View Action Recognition

SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization

Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

Inter-Class and Inter-Domain Semantic Augmentation for Domain Generalization

Gradient-Map-Guided Adaptive Domain Generalization for Cross Modality MRI Segmentation

Generalizable Cross-modality Medical Image Segmentation Via Style Augmentation and Dual Normalization

Towards Domain Generalization for Multi-view 3D Object Detection in Bird-Eye-View

Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering

Multisource Collaborative Domain Generalization for Cross-Scene Remote Sensing Image Classification

Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks

Domain Generalized Fundus Image Segmentation Via Dual-Level Mixing

Robust Domain Generalization for Multi-modal Object Recognition