Abstract:Temporal sentence grounding in videos is a crucial task in vision-language learning. Its goal is retrieving a video segment from an untrimmed video that semantically corresponds to a natural language query. A video usually contains multiple semantic events, which are rarely isolated. They tend to be temporally ordered and semantically correlated (e.g., some event is often the precursor of another event). To precisely localize a semantic moment from a video, it is critical to effectively extract and aggregate multi-granularity contextual information, including the fine-grained local context around the moment-related video segment (in short snippet-level) and coarse-grained semantic correlation (in segment-level). Additionally, a second main insight in this work is that the above context aggregation should be favorably guided by the queries, rather than fully query-agnostic. Putting above ideas together, we here present a new network that does language-guided multi-granularity context aggregation. It is comprised of two major modules. The core of the first module is a novel language-guided temporal adaptive convolution (LTAC) devised to extract fine-grained information over video snippets around the ground-truth video segment. It decomposes a convolution into two channel-oriented / temporal-oriented ones. In particular, the convolutional channels are supposed to be more susceptible to queries, thus we learn to generate a dynamic channel-oriented kernel with respect to the querying sentence. As a second module, we propose a language-guided global relation block (LGRB) that extracts video-level context. It augments the contextual feature by using a multi-scale temporal attention that tackles the scale variation of ground-truth video segments, and a multi-modal semantic attention that relies on syntactic of the query. For the validation purpose, we have conducted comprehensive experiments on two popularly-adopted video benchmarks (i.e., ActivityNet Captions and Charades-STA). All experimental results and ablation studies have clearly corroborated the effectiveness of our model designs, outstripping prior state-of-the-art methods in terms of major performance metrics for the task.

DCT-net: A Deep Co-Interactive Transformer Network for Video Temporal Grounding

Temporal-enhanced Cross-modality Fusion Network for Video Sentence Grounding.

Language-Guided Multi-Granularity Context Aggregation for Temporal Sentence Grounding

Cross-Graph Transformer Network for Temporal Sentence Grounding

End-to-end Multi-modal Video Temporal Grounding

BiC-Net: Learning Efficient Spatio-Temporal Relation for Text-Video Retrieval

Hierarchical Local-Global Transformer for Temporal Sentence Grounding.

DTT-CGINet: A Dual Temporal Transformer Network with Multi-Scale Contour-Guided Graph Interaction for Change Detection

PTAN: Principal Token-aware Adjacent Network for Compositional Temporal Grounding

SDN: Semantic Decoupling Network for Temporal Language Grounding

Context-aware Biaffine Localizing Network for Temporal Sentence Grounding

Multi-stage Aggregated Transformer Network for Temporal Language Localization in Videos

Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding

Training-free Video Temporal Grounding using Large-scale Pre-trained Models

Context-aware network with foreground recalibration for grounding natural language in video

Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos.

SLTFNet: A Spatial and Language-Temporal Tensor Fusion Network for Video Moment Retrieval

Multi-Scale Temporal Difference Transformer for Video-Text Retrieval

Proposal-Free Video Grounding with Contextual Pyramid Network

Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning

Explore Inter-contrast between Videos via Composition for Weakly Supervised Temporal Sentence Grounding.