LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

Haoning Wu,Dongxu Li,Bei Chen,Junnan Li

2024-07-23

Abstract:Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

Computer Vision and Pattern Recognition,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is:

Current large multimodal models (LMMs) lack sufficient benchmarks for handling long-term video understanding. Despite progress in handling long inputs, existing video understanding benchmarks mainly focus on single-frame bias, meaning that results do not significantly improve when models process more frames. To address this issue, the paper proposes LONGVIDEO BENCH, a benchmark designed to evaluate the ability to understand long-term videos.

Specifically, LONGVIDEO BENCH includes a series of network-collected videos of different lengths and their subtitles, aiming to comprehensively assess the performance of LMMs in long-term multimodal understanding. This benchmark overcomes single-frame bias by introducing the "referential reasoning" task, which requires models to accurately retrieve and reason detailed multimodal information from long inputs. Additionally, the paper validates the performance of different models on this benchmark through extensive experiments and finds that even the most advanced proprietary models face challenges, while the performance gap is larger for open-source models.

Therefore, the main goal of this paper is to provide an effective and challenging benchmark to drive the development of future long-term context multimodal models.