Abstract:Video temporal grounding aims to identify video segments within untrimmed videos that are most relevant to a given natural language query. Existing video temporal localization models rely on specific datasets for training and have high data collection costs, but they exhibit poor generalization capability under the across-dataset and out-of-distribution (OOD) settings. In this paper, we propose a Training-Free Video Temporal Grounding (TFVTG) approach that leverages the ability of pre-trained large models. A naive baseline is to enumerate proposals in the video and use the pre-trained visual language models (VLMs) to select the best proposal according to the vision-language alignment. However, most existing VLMs are trained on image-text pairs or trimmed video clip-text pairs, making it struggle to (1) grasp the relationship and distinguish the temporal boundaries of multiple events within the same video; (2) comprehend and be sensitive to the dynamic transition of events (the transition from one event to another) in the video. To address these issues, we propose leveraging large language models (LLMs) to analyze multiple sub-events contained in the query text and analyze the temporal order and relationships between these events. Secondly, we split a sub-event into dynamic transition and static status parts and propose the dynamic and static scoring functions using VLMs to better evaluate the relevance between the event and the description. Finally, for each sub-event description, we use VLMs to locate the top-k proposals and leverage the order and relationships between sub-events provided by LLMs to filter and integrate these proposals. Our method achieves the best performance on zero-shot video temporal grounding on Charades-STA and ActivityNet Captions datasets without any training and demonstrates better generalization capabilities in cross-dataset and OOD settings.

Zero-Shot Video Grounding for Automatic Video Understanding in Sustainable Smart Cities

Zero-Shot Video Grounding With Pseudo Query Lookup and Verification

SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding

Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems

Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding

GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection

Language-free Training for Zero-shot Video Grounding

VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT

Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding

Training-free Video Temporal Grounding using Large-scale Pre-trained Models

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

Unsupervised Temporal Video Grounding with Deep Semantic Clustering

DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization.

End-to-End Modeling Via Information Tree for One-Shot Natural Language Spatial Video Grounding

VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning

Zero-Shot Scene Classification for High Spatial Resolution Remote Sensing Images

Video Grounding and Its Generalization

VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs

VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding

Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)

Agent3D-Zero: An Agent for Zero-shot 3D Understanding