Abstract:Video grounding is a fundamental problem in multimodal content understanding, aiming to localize specific natural language queries in an untrimmed video. However, current video grounding datasets merely focus on simple events and are either limited to shorter videos or brief sentences, which hinders the model from evolving toward stronger multimodal understanding capabilities. To address these limitations, we present a large-scale video grounding dataset named SynopGround, in which more than 2800 hours of videos are sourced from popular TV dramas and are paired with accurately localized human-written synopses. Each paragraph in the synopsis serves as a language query and is manually annotated with precise temporal boundaries in the long video. These paragraph queries are tightly correlated to each other and contain a wealth of abstract expressions summarizing video storylines and specific descriptions portraying event details, which enables the model to learn multimodal perception on more intricate concepts over longer context dependencies. Based on the dataset, we further introduce a more complex setting of video grounding dubbed Multi-Paragraph Video Grounding (MPVG), which takes as input multiple paragraphs and a long video for grounding each paragraph query to its temporal interval. In addition, we propose a novel Local-Global Multimodal Reasoner (LGMR) to explicitly model the local-global structures of long-term multimodal inputs for MPVG. Our method provides an effective baseline solution to the multi-paragraph video grounding problem. Extensive experiments verify the proposed model's effectiveness as well as its superiority in long-term multi-paragraph video grounding over prior state-of-the-arts. Dataset and code are publicly available. Project page: <a class="link-external link-https" href="https://synopground.github.io/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper aims to address several key issues in the field of video grounding, particularly by proposing a new solution to the limitations of current datasets. Specifically, the issues raised in the paper include: 1. **Limitations of existing datasets**: Most existing video grounding datasets focus only on simple events and are limited to shorter videos or brief sentence queries, which restricts the development of models' ability to understand complex multimodal content. 2. **Semantic ambiguity**: Shorter query descriptions can easily lead to semantic ambiguity, meaning a single query might correspond to multiple video segments, which is detrimental to model learning. 3. **Lack of support for complex events and abstract concepts**: Existing benchmark datasets mainly focus on low-level descriptions of visible activities, neglecting the importance of more complex events and abstract concepts. 4. **Understanding of long-term context**: Current technologies fail to meet practical needs such as automatically integrating raw footage into script-based coherent stories, which requires the ability to associate long videos with complex descriptions. To address the above issues, the main contributions of the paper are as follows: - A large-scale video grounding dataset named SynopGround is proposed, which contains over 2800 hours of TV series videos, accompanied by manually written, precisely annotated plot summaries. - Based on this dataset, a more challenging video grounding task is introduced—Multi-Paragraph Video Grounding (MPVG), which aims to locate the video segments corresponding to each paragraph in a given multi-paragraph plot summary and a long video. - A new Local-Global Multimodal Reasoner (LGMR) is designed to explicitly model the local-global structure of long temporal multimodal inputs and perform iterative cross-modal reasoning within and between these structures. Through these contributions, the paper not only provides a high-quality dataset to support researchers in exploring long-term context video grounding issues but also offers an effective baseline method as a starting point for solving these problems.

SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses

Dense Events Grounding in Video.

Semi-supervised Video Paragraph Grounding with Contrastive Encoder

Dual-task Mutual Reinforcing Embedded Joint Video Paragraph Retrieval and Grounding

Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding

DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization.

Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning

Temporal Sentence Grounding in Videos with Fine-Grained Multimodal Correlation

Multi-Sentence Grounding for Long-term Instructional Video

Video Grounding and Its Generalization

A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metric

Grounded Video Description

Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

End-to-end dense video grounding via parallel regression

Hierarchical Semantic Correspondence Networks for Video Paragraph Grounding

Efficient Video Grounding with Which-Where Reading Comprehension

Look Closer to Ground Better: Weakly-Supervised Temporal Grounding of Sentence in Video

Video-Guided Curriculum Learning for Spoken Video Grounding

End-to-end Multi-modal Video Temporal Grounding

Unsupervised Temporal Video Grounding with Deep Semantic Clustering