Abstract:Temporal Sentence Grounding in Videos (TSGV), which aims to ground a natural language sentence that indicates complex human activities in an untrimmed video, has drawn widespread attention over the past few years. However, recent studies have found that current benchmark datasets may have obvious moment annotation biases, enabling several simple baselines even without training to achieve state-of-the-art (SOTA) performance. In this paper, we take a closer look at existing evaluation protocols for TSGV, and find that both the prevailing dataset splits and evaluation metrics are the devils that lead to untrustworthy benchmarking. Therefore, we propose to re-organize the two widely-used datasets, making the ground-truth moment distributions different in the training and test splits, i.e., out-of-distribution (OOD) test. Meanwhile, we introduce a new evaluation metric “dR@n,IoU@m” that discounts the basic recall scores especially with small IoU thresholds, so as to alleviate the inflating evaluation caused by biased datasets with a large proportion of long ground-truth moments. New benchmarking results indicate that our proposed evaluation protocols can better monitor the research progress in TSGV. Furthermore, we propose a novel causality-based Multi-branch Deconfounding Debiasing (MDD) framework for unbiased moment prediction. Specifically, we design a multi-branch deconfounder to eliminate the effects caused by multiple confounders with causal intervention. In order to help the model better align the semantics between sentence queries and video moments, we enhance the representations during feature encoding. Specifically, for textual information, the query is parsed into several verb-centered phrases to obtain a more fine-grained textual feature. For visual information, the positional information has been decomposed from the moment features to enhance the representations of moments with diverse locations. Extensive experiments demonstrate that our proposed approach can achieve competitive results among existing SOTA approaches and outperform the base model with great gains.

Debiased Video-Text Retrieval via Soft Positive Sample Calibration

Robust Video-Text Retrieval via Noisy Pair Calibration

Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention

Reducing the Vision and Language Bias for Temporal Sentence Grounding.

Sample-Specific Debiasing for Better Image-Text Models

Using Bag of Visual Words for Video Retrieval Calibration

Bi-calibration Networks for Weakly-Supervised Video Representation Learning

UATVR: Uncertainty-Adaptive Text-Video Retrieval

Improving semantic video retrieval models by training with a relevance-aware online mining strategy

Learning Sample Importance for Cross-Scenario Video Temporal Grounding

Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning

A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach.

Selective Query-guided Debiasing for Video Corpus Moment Retrieval

QD-VMR: Query Debiasing with Contextual Understanding Enhancement for Video Moment Retrieval

TeachText: CrossModal Generalized Distillation for Text-Video Retrieval

Curriculum Multi-Negative Augmentation for Debiased Video Grounding.

Bias-Conflict Sample Synthesis and Adversarial Removal Debias Strategy for Temporal Sentence Grounding in Video

Exploiting Web Images for Semantic Video <newline/>indexing Via Robust Sample-Specific Loss

Towards Robust Text-Prompted Semantic Criterion for In-the-Wild Video Quality Assessment

Suppressing Biased Samples for Robust VQA

Reducing 0s Bias in Video Moment Retrieval with a Circular Competence-Based Captioner