Abstract:Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos. However, they perform almost the same as random on grounding text queries in long and complicated videos, having little ability to understand and reason about temporal information, which is the most fundamental difference between videos and images. In this paper, we propose HawkEye, one of the first video-text LLMs that can perform temporal video grounding in a fully text-to-text manner. To collect training data that is applicable for temporal video grounding, we construct InternVid-G, a large-scale video-text corpus with segment-level captions and negative spans, with which we introduce two new time-aware training objectives to video-text LLMs. We also propose a coarse-grained method of representing segments in videos, which is more robust and easier for LLMs to learn and follow than other alternatives. Extensive experiments show that HawkEye is better at temporal video grounding and comparable on other video-text tasks with existing video-text LLMs, which verifies its superior video-text multi-modal understanding abilities.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the insufficient ability to perform temporal video grounding in long - form videos. Existing video - text large language models (video - text LLMs) perform well when dealing with simple videos, but their performance in temporal video grounding for text queries in long and complex videos is almost no different from random performance, especially with limited ability in understanding and reasoning about temporal information. Temporal information is one of the most fundamental differences between videos and images, but most video - text LLMs overlook this. Specifically, the paper points out that although existing models can hold conversations and answer questions on short and simple videos, they perform poorly for understanding long - form videos, which are important components in daily life, such as movies, tutorials, and documentaries. This is because the computer first needs to understand the basic content of the video and then understand the sequence of multiple events in the video, which is especially difficult in long - form videos. Therefore, the paper proposes HawkEye, a model aimed at enhancing the temporal video grounding ability of existing video - text LLMs through training for long - form videos. The main contributions of HawkEye include: 1. **Well - designed video segment representation format**: The paper proposes a coarse - grained method to represent video segments. This method is easier for LLMs to learn and follow, rather than directly generating the start frame, end frame, or timestamp of video segments. 2. **Construction of a large - scale temporally - aware instruction training dataset**: The paper constructs InternVid - G, a large - scale video - text corpus containing 715,000 segment - level subtitles and negative segments, which is suitable for constructing training samples for temporal video grounding. Through these improvements, HawkEye can perform temporal video grounding tasks in a completely text - to - text manner and remain competitive in other video - text tasks. Experimental results show that HawkEye's performance in the temporal video grounding task is significantly better than the random level, verifying its superior ability in video - text multi - modal understanding.

HawkEye: Training Video-Text LLMs for Grounding Text in Videos

Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

LLM4VG: Large Language Models Evaluation for Video Grounding

Training-free Video Temporal Grounding using Large-scale Pre-trained Models

VTimeLLM: Empower LLM to Grasp Video Moments

VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos

ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Models

ST-LLM: Large Language Models Are Effective Temporal Learners

Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos

TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning

VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding

VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format

Temporal Reasoning Transfer from Text to Video

Infusing Environmental Captions for Long-Form Video Language Grounding

Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Temporal Sentence Grounding in Videos with Fine-Grained Multimodal Correlation

Language-free Training for Zero-shot Video Grounding

Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos

TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models

Video LLMs for Temporal Reasoning in Long Videos