Abstract:This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

What problem does this paper attempt to address?

### The Problem Addressed by the Paper This paper aims to address the issues present in evaluating the true understanding and reasoning capabilities of current Multimodal Large Language Models (MLLMs). Specifically, the paper proposes a more robust and challenging benchmark test—MMMU-Pro, to compensate for the shortcomings of existing benchmarks such as MMMU. #### Existing Issues 1. **Text Dependency**: Some existing benchmark questions can be answered by pure text models without relying on image information. 2. **Option Guessing**: Even when image information is required, models can sometimes find the correct answer through subtle clues in the options rather than truly understanding the multimodal information. 3. **Lack of Real-World Simulation**: Existing benchmarks fail to adequately simulate the complexity and diversity of multimodal tasks in the real world. #### Solution To overcome these issues, the paper introduces MMMU-Pro, which constructs a more rigorous evaluation tool through the following three steps: 1. **Filtering Questions**: Exclude questions that can be answered by pure text models, ensuring that the remaining questions genuinely require multimodal understanding. 2. **Enhancing Options**: Increase the candidate options from 4 to 10, reducing the likelihood of guessing the correct answer. 3. **Introducing Visual Input Settings**: Embed questions in screenshots or photos without providing explicit text, testing whether the model can both "see" and "read," simulating the human ability to handle complex scenarios. #### Experimental Results The experimental results show that all tested models performed significantly worse on MMMU-Pro compared to the original MMMU benchmark, with performance drops ranging from 16.8% to 26.9%. Further analysis indicates that while Chain of Thought (CoT) prompts generally improve performance, simply using OCR prompts has little impact on most models. These results highlight the limitations of current state-of-the-art multimodal models in true multimodal understanding and reasoning. #### Significance MMMU-Pro not only provides a more stringent evaluation tool but also offers valuable directions for future research in multimodal AI, particularly in handling complex, mixed-format inputs and enhancing multimodal reasoning capabilities.

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Published at NeurIPS 2024 Track Datasets and Benchmarks)

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

MULTI: Multimodal Understanding Leaderboard with Text and Images

MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation

MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks

MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification

MMR: Evaluating Reading Ability of Large Multimodal Models

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

MMBench: Is Your Multi-modal Model an All-around Player?

MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

MuMA-ToM: Multi-modal Multi-Agent Theory of Mind

PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathology

MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks

CMMLU: Measuring massive multitask language understanding in Chinese