Abstract:Spatio-Temporal Video Grounding (STVG) aims at localizing the spatio-temporal tube of a specific object in an untrimmed video given a free-form natural language query. As the annotation of tubes is labor intensive, researchers are motivated to explore weakly supervised approaches in recent works, which usually results in significant performance degradation. To achieve a less expensive STVG method with acceptable accuracy, this work investigates the "single-frame supervision" paradigm that requires a single frame labeled with a bounding box within the temporal boundary of the fully supervised counterpart as the supervisory signal. Based on the characteristics of the STVG problem, we propose a Two-Stage Multiple Instance Learning (T-SMILE) method, which creates pseudo labels by expanding the annotated frame to its contextual frames, thereby establishing a fully-supervised problem to facilitate further model training. The innovations of the proposed method are three-folded, including 1) utilizing multiple instance learning to dynamically select instances in positive bags for the recognition of starting and ending timestamps, 2) learning highly discriminative query features by incorporating spatial prior constraints in cross-attention, and 3) designing a curriculum learning-based strategy that iterative assigns dynamic weights to spatial and temporal branches, thereby gradually adapting to the learning branch with larger difficulty. To facilitate future research on this task, we also contribute a large-scale benchmark containing 12,469 videos on complex scenes with single-frame annotation. The extensive experiments on two benchmarks demonstrate that T-SMILE significantly outperforms all weakly-supervised methods. Remarkably, it also performs better than some fully-supervised methods associated with much more annotation labor costs. The dataset and codes are available at https://github.com/qumengxue/T-SMILE.

CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Grounding

Human-centric Spatio-Temporal Video Grounding with Visual Transformers

STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation.

End-to-end Multi-modal Video Temporal Grounding

Context-Guided Spatio-Temporal Video Grounding

Collaborative Static and Dynamic Vision-Language Streams for Spatio-Temporal Video Grounding

Matching and Localizing: A Simple yet Effective Framework for Human-Centric Spatio-Temporal Video Grounding

Human-centric Spatio-Temporal Video Grounding Via the Combination of Mutual Matching Network and TubeDETR

STVGFormer: Spatio-Temporal Video Grounding with Static-Dynamic Cross-Modal Understanding

Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentences

Weakly-Supervised Spatio-Temporal Video Grounding with Variational Cross-Modal Alignment

Decoupled Spatial Temporal Graphs for Generic Visual Grounding

STVGFormer

STVGBert - A Visual-linguistic Transformer Based Framework for Spatio-temporal Video Grounding.

Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding

Unsupervised Temporal Video Grounding with Deep Semantic Clustering

Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding

CoSTA: Co-training Spatial-Temporal Attention for Blind Video Quality Assessment

Single-Frame Supervision for Spatio-Temporal Video Grounding