Abstract:Temporal Sentence Grounding in Videos (TSGV), which aims to ground a natural language sentence that indicates complex human activities in an untrimmed video, has drawn widespread attention over the past few years. However, recent studies have found that current benchmark datasets may have obvious moment annotation biases, enabling several simple baselines even without training to achieve state-of-the-art (SOTA) performance. In this paper, we take a closer look at existing evaluation protocols for TSGV, and find that both the prevailing dataset splits and evaluation metrics are the devils that lead to untrustworthy benchmarking. Therefore, we propose to re-organize the two widely-used datasets, making the ground-truth moment distributions different in the training and test splits, i.e., out-of-distribution (OOD) test. Meanwhile, we introduce a new evaluation metric “dR@n,IoU@m” that discounts the basic recall scores especially with small IoU thresholds, so as to alleviate the inflating evaluation caused by biased datasets with a large proportion of long ground-truth moments. New benchmarking results indicate that our proposed evaluation protocols can better monitor the research progress in TSGV. Furthermore, we propose a novel causality-based Multi-branch Deconfounding Debiasing (MDD) framework for unbiased moment prediction. Specifically, we design a multi-branch deconfounder to eliminate the effects caused by multiple confounders with causal intervention. In order to help the model better align the semantics between sentence queries and video moments, we enhance the representations during feature encoding. Specifically, for textual information, the query is parsed into several verb-centered phrases to obtain a more fine-grained textual feature. For visual information, the positional information has been decomposed from the moment features to enhance the representations of moments with diverse locations. Extensive experiments demonstrate that our proposed approach can achieve competitive results among existing SOTA approaches and outperform the base model with great gains.

Temporal Sentence Grounding in Streaming Videos

A Survey on Temporal Sentence Grounding in Videos

Temporal Sentence Grounding in Videos: A Survey and Future Directions

Temporal Sentence Grounding in Videos with Fine-Grained Multimodal Correlation

You Can Ground Earlier Than See: an Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos

A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metric

CMGN: Cross-Modal Grounding Network for Temporal Sentence Retrieval in Video.

Fine-Grained Text-to-Video Temporal Grounding from Coarse Boundary

Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video

Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding

Few-Shot Temporal Sentence Grounding Via Memory-Guided Semantic Learning

A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach.

Conditional Video Diffusion Network for Fine-grained Temporal Sentence Grounding

Tracking Objects and Activities with Attention for Temporal Sentence Grounding

Towards Debiasing Temporal Sentence Grounding in Video

Unsupervised Temporal Video Grounding with Deep Semantic Clustering

Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos

Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding.

Memory-Guided Semantic Learning Network for Temporal Sentence Grounding

Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network