Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning

Long Qian,Juncheng Li,Yu Wu,Yaobo Ye,Hao Fei,Tat-Seng Chua,Yueting Zhuang,Siliang Tang

2024-06-02

Abstract:Large Language Models (LLMs) demonstrate remarkable proficiency in comprehending and handling text-based tasks. Many efforts are being made to transfer these attributes to video modality, which are termed Video-LLMs. However, existing Video-LLMs can only capture the coarse-grained semantics and are unable to effectively handle tasks related to comprehension or localization of specific video segments. In light of these challenges, we propose Momentor, a Video-LLM capable of accomplishing fine-grained temporal understanding tasks. To support the training of Momentor, we design an automatic data generation engine to construct Moment-10M, a large-scale video instruction dataset with segment-level instruction data. We train Momentor on Moment-10M, enabling it to perform segment-level reasoning and localization. Zero-shot evaluations on several tasks demonstrate that Momentor excels in fine-grained temporally grounded comprehension and localization.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper primarily addresses the limitations of existing Video Large Language Models (Video-LLMs) in handling video segment-level tasks, proposing a new approach. Specifically, while existing Video-LLMs can understand the overall semantics of a video to some extent, they fall short in the following areas:

1. **Insufficient effectiveness of temporal representation**: Existing models encode each sampled frame independently and perform feature projection but do not retain precise temporal information. This results in their inability to effectively represent and output specific timestamps.
2. **Insufficient segment-level modeling capability**: Existing models mainly focus on global visual semantics, neglecting the modeling of segment-level semantics and relationships. Therefore, these models perform poorly when dealing with specific segments in long videos.

To address these issues, the paper proposes **Momentor**, a video large language model with fine-grained temporal perception and segment-level reasoning capabilities. To enhance temporal modeling capabilities, the paper introduces innovations in the model architecture and training methods, incorporating a **Temporal Perception Module** (TPM) and adding a **Grounded Event-Sequence Modeling** phase during training. Additionally, the paper constructs a large-scale video instruction dataset **Moment-10M** to support model training.

With these improvements, Momentor excels in multiple tasks involving precise temporal localization, such as temporal localization, dense captioning, action segmentation, and highlight moment retrieval.