Abstract:As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations (i.e., moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel M ulti- M odal C ross- D omain A lignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; and (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Challenge Huawei challenge: Fusing multimodal features with deep neural networks for Mobile Video Annotation

Emotion Recognition in Videos via Fusing Multimodal Features.

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Msr asia msm at activitynet challenge 2016

A Novel Deep Multi-Modal Feature Fusion Method for Celebrity Video Identification

Fudan-NJUST at MediaEval 2014: Violent Scenes Detection Using Deep Neural Networks.

Fusing Multi-Stream Deep Networks for Video Classification

Fudan-Huawei at MediaEval 2015: Detecting Violent Scenes and Affective Impact in Movies with Deep Learning.

CUHK ETHZ SIAT Submission to ActivityNet Challenge 2016

Deep Multimodal Learning: An Effective Method for Video Classification

Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification.

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification

MFVC: Urban Traffic Scene Video Caption Based on Multimodal Fusion

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

DeepMEF

Hybrid Improvements in Multimodal Analysis for Deep Video Understanding

Aggregating Frame-level Features for Large-Scale Video Classification

Research on Feature Extraction and Multimodal Fusion of Video Caption Based on Deep Learning