Abstract:In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. 900 videos with a total of 254 hours are manually selected and annotated by repeatedly viewing all the video content, resulting in 2,700 question-answer pairs. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data. Project Page: <a class="link-external link-https" href="https://video-mme.github.io" rel="external noopener nofollow">this https URL</a>

A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases

Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models

From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

An Evaluation of GPT-4V and Gemini in Online VQA

Gemini: A Family of Highly Capable Multimodal Models

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models

Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

An In-depth Look at Gemini's Language Abilities

A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study

VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Gemini Pro Defeated by GPT-4V: Evidence from Education

Capabilities of Gemini Models in Medicine

GPT-4V(ision) is a Generalist Web Agent, if Grounded

VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework

Comparing Vision-Capable Models, GPT-4 and Gemini, With GPT-3.5 on Taiwan's Pulmonologist Exam