MovieChat+: Question-aware Sparse Memory for Long Video Question Answering

Enxin Song,Wenhao Chai,Tian Ye,Jenq-Neng Hwang,Xi Li,Gaoang Wang

2024-04-26

Abstract:Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, and they only perform well on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose MovieChat to overcome these challenges. We lift pre-trained multi-modal large language models for understanding long videos without incorporating additional trainable temporal modules, employing a zero-shot approach. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video, 2K temporal grounding labels, and 14K manual annotations for validation of the effectiveness of our method. The code along with the dataset can be accessed via the following

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of long video understanding, particularly for ultra-long videos exceeding 10,000 frames. Current methods face major challenges when dealing with long videos, including high computational complexity, large memory costs, and additional difficulties arising from long-term temporal connections. Specifically:

1. **Computational Complexity and Memory Costs**: Existing methods either rely on complex spatiotemporal modules or heavily depend on additional perceptual tools to obtain the temporal information needed for video understanding. These methods significantly increase computational complexity and memory consumption when processing long videos.

2. **Lack of Standardized Benchmarks**: There is currently a lack of a standardized benchmark to evaluate the performance of these systems.

To address these issues, the paper proposes MovieChat and its improved version, MovieChat+. MovieChat leverages pre-trained Multimodal Large Language Models (MLLMs) and adopts a zero-shot approach to understand long videos without introducing additional trainable temporal modules. Additionally, inspired by the Atkinson-Shiffrin memory model, the paper designs a memory mechanism that combines short-term memory and long-term memory, and further enhances the memory consolidation mechanism for vision-question matching in MovieChat+, thereby significantly improving prediction accuracy.

The paper also releases a benchmark dataset, MovieChat-1K, which includes 1,000 long videos and 14,000 manually annotated question-answer pairs, to validate the effectiveness of the proposed methods. Furthermore, the paper extends the dataset by adding 2,000 temporal tags to better evaluate the model's ability to identify relevant video segments. Overall, the paper makes significant progress in long video understanding and question-answering tasks by introducing effective memory mechanisms.