Abstract:Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3,763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6,678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

ViLCo-Bench: VIdeo Language COntinual learning Benchmark

LVBench: An Extreme Long Video Understanding Benchmark

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition

Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding

TVBench: Redesigning Video-Language Evaluation

VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning

BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos

SAVEn-Vid: Synergistic Audio-Visual Integration for Enhanced Understanding in Long Video Context

VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model

CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection

VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models