Abstract:Video moment retrieval targets at retrieving a golden moment in a video for a given natural language query. The main challenges of this task include 1) the requirement of accurately localizing (i.e., the start time and the end time of) the relevant moment in an untrimmed video stream, and 2) bridging the semantic gap between textual query and video contents. To tackle those problems, early approaches adopt the sliding window or uniform sampling to collect video clips first and then match each clip with the query to identify relevant clips. Obviously, these strategies are time-consuming and often lead to unsatisfied accuracy in localization due to the unpredictable length of the golden moment. To avoid the limitations, researchers recently attempt to directly predict the relevant moment boundaries without the requirement to generate video clips first. One mainstream approach is to generate a multimodal feature vector for the target query and video frames (e.g., concatenation) and then use a regression approach upon the multimodal feature vector for boundary detection. Although some progress has been achieved by this approach, we argue that those methods have not well captured the cross-modal interactions between the query and video frames. In this paper, we propose an Attentive Cross-modal Relevance Matching (ACRM) model which predicts the temporal boundaries based on an interaction modeling between two modalities. In addition, an attention module is introduced to automatically assign higher weights to query words with richer semantic cues, which are considered to be more important for finding relevant video contents. Another contribution is that we propose an additional predictor to utilize the internal frames in the model training to improve the localization accuracy. Extensive experiments on two public datasets TACoS and Charades-STA demonstrate the superiority of our method over several state-of-the-art methods. Ablation studies have been also conducted to exami-e the effectiveness of different modules in our ACRM model.

Length-Aware DETR for Robust Moment Retrieval

Background-aware Moment Detection for Video Moment Retrieval

Query-Dependent Video Representation for Moment Retrieval and Highlight Detection

BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos

MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction

Saliency-Guided DETR for Moment Retrieval and Highlight Detection

TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection

MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer

MomentDiff: Generative Video Moment Retrieval from Random to Real

Language-based Audio Moment Retrieval

2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval

Attentive Moment Retrieval in Videos

QD-VMR: Query Debiasing with Contextual Understanding Enhancement for Video Moment Retrieval

Selective Query-guided Debiasing for Video Corpus Moment Retrieval

MVMR: A New Framework for Evaluating Faithfulness of Video Moment Retrieval against Multiple Distractors

Adversarial Video Moment Retrieval by Jointly Modeling Ranking and Localization

Video Moment Retrieval with Noisy Labels

The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval

Moment Detection in Long Tutorial Videos

GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features

Frame-Wise Cross-Modal Matching for Video Moment Retrieval