Abstract:Video moment retrieval targets at retrieving a golden moment in a video for a given natural language query. The main challenges of this task include 1) the requirement of accurately localizing (i.e., the start time and the end time of) the relevant moment in an untrimmed video stream, and 2) bridging the semantic gap between textual query and video contents. To tackle those problems, early approaches adopt the sliding window or uniform sampling to collect video clips first and then match each clip with the query to identify relevant clips. Obviously, these strategies are time-consuming and often lead to unsatisfied accuracy in localization due to the unpredictable length of the golden moment. To avoid the limitations, researchers recently attempt to directly predict the relevant moment boundaries without the requirement to generate video clips first. One mainstream approach is to generate a multimodal feature vector for the target query and video frames (e.g., concatenation) and then use a regression approach upon the multimodal feature vector for boundary detection. Although some progress has been achieved by this approach, we argue that those methods have not well captured the cross-modal interactions between the query and video frames. In this paper, we propose an Attentive Cross-modal Relevance Matching (ACRM) model which predicts the temporal boundaries based on an interaction modeling between two modalities. In addition, an attention module is introduced to automatically assign higher weights to query words with richer semantic cues, which are considered to be more important for finding relevant video contents. Another contribution is that we propose an additional predictor to utilize the internal frames in the model training to improve the localization accuracy. Extensive experiments on two public datasets TACoS and Charades-STA demonstrate the superiority of our method over several state-of-the-art methods. Ablation studies have been also conducted to exami-e the effectiveness of different modules in our ACRM model.

Cross-Modal Video Moment Retrieval with Spatial and Language-Temporal Attention.

Attentive Moment Retrieval in Videos

Cross-modal Moment Localization in Videos.

Collaborative Spatial-Temporal Interaction for Language-Based Moment Retrieval

SLTFNet: A Spatial and Language-Temporal Tensor Fusion Network for Video Moment Retrieval

Cross-Modal Interaction Network for Video Moment Retrieval.

Frame-Wise Cross-Modal Matching for Video Moment Retrieval

Cross-modal Video Moment Retrieval Based on Visual-Textual Relationship Alignment

Cross-modal Video Moment Retrieval Based on Enhancing Significant Features

Moment Retrieval via Cross-Modal Interaction Networks with Query Reconstruction

Language Guided Networks for Cross-modal Moment Retrieval

Local-enhanced Interaction for Temporal Moment Localization

Video Moment Retrieval Via Comprehensive Relation-Aware Network

Cross-Modal Dynamic Networks for Video Moment Retrieval With Text Query

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

A Multi-modal Fusion Algorithm for Cross-modal Video Moment Retrieval

Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval

Video Description with Spatial-Temporal Attention.

STRONG: Spatio-Temporal Reinforcement Learning for Cross-Modal Video Moment Localization

Triadic Temporal-Semantic Alignment for Weakly-Supervised Video Moment Retrieval

Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language