Abstract:In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. 900 videos with a total of 254 hours are manually selected and annotated by repeatedly viewing all the video content, resulting in 2,700 question-answer pairs. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data. Project Page: <a class="link-external link-https" href="https://video-mme.github.io" rel="external noopener nofollow">this https URL</a>

An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM

LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models

Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

Modular Blind Video Quality Assessment

Enhancing Blind Video Quality Assessment with Rich Quality-aware Features

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

MT-VQA: A Multi-task Approach for Quality Assessment of Short-form Videos

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding

BoViLA: Bootstrapping Video-Language Alignment via LLM-Based Self-Questioning and Answering

Demonstrative Instruction Following in Multimodal LLMs Via Integrating Low-Rank Adaptation with Ensemble Learning

Visual Context Window Extension: A New Perspective for Long Video Understanding

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens

HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data

VideoQA in the Era of LLMs: An Empirical Study

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

MLVU: Benchmarking Multi-task Long Video Understanding

Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models

Understanding Long Videos with Multimodal Language Models