Towards Event-oriented Long Video Understanding

Yifan Du,Kun Zhou,Yuqi Huo,Yifan Li,Wayne Xin Zhao,Haoyu Lu,Zijia Zhao,Bingning Wang,Weipeng Chen,Ji-Rong Wen
2024-06-20
Abstract:With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at <a class="link-external link-https" href="https://github.com/RUCAIBox/Event-Bench" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition,Computation and Language,Multimedia
What problem does this paper attempt to address?
The paper primarily aims to address two core issues in the field of video understanding: 1. **Shortcut Bias in Datasets**: Existing video understanding datasets lack rich events, allowing models to infer answers by looking at only a few frames without watching the entire video. This phenomenon reduces the reliability of evaluations. 2. **Scarcity of Event-Dense Video Instructions**: To train video multimodal large language models (Video MLLMs) capable of understanding complex events, a large amount of event-dense video instructions is required. However, annotating such data is costly. To address these issues, the paper proposes two main contributions: - **Event-Bench for Long Video Understanding**: This benchmark includes six event-related tasks and 2,190 test instances, aiming to comprehensively evaluate video event understanding capabilities. - **Video Instruction Merging (VIM) Method**: A low-cost method that generates new event-dense video instructions by merging similar video instructions from existing datasets to enhance the capabilities of video MLLMs. Experimental results show that models trained using the VIM method significantly outperform existing open-source models and some proprietary models on the Event-Bench.