MULTI: Multimodal Understanding Leaderboard with Text and Images

Zichen Zhu,Yang Xu,Lu Chen,Jingkai Yang,Yichuan Ma,Yiming Sun,Hailin Wen,Jiaqi Liu,Jinyu Cai,Yingzi Ma,Situo Zhang,Zihan Zhao,Liangtai Sun,Kai Yu

2024-02-20

Abstract:Rapid progress in multimodal large language models (MLLMs) highlights the need to introduce challenging yet realistic benchmarks to the academic community, while existing benchmarks primarily focus on understanding simple natural images and short context. In this paper, we present MULTI as a cutting-edge benchmark for evaluating MLLMs on understanding complex tables and images, and reasoning with long context. MULTI provides multimodal inputs and requires responses that are either precise or open-ended, reflecting real-life examination styles. MULTI includes over 18,000 questions and challenges MLLMs with a variety of tasks, ranging from formula derivation to image detail analysis and cross-modality reasoning. We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend, with more than 4,500 external knowledge context pieces. Our evaluation indicates significant potential for MLLM advancement, with GPT-4V achieving a 63.7% accuracy rate on MULTI, in contrast to other MLLMs scoring between 28.5% and 55.3%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue that existing multimodal large language model (MLLM) evaluation benchmarks are overly simplistic, primarily focusing on understanding simple natural images and short texts, and failing to comprehensively assess the models' capabilities in handling complex tables, long texts, and cross-modal reasoning tasks. To this end, the paper proposes a new benchmark named MULTI, aimed at evaluating MLLMs' abilities in understanding complex tables and images, as well as processing long texts. MULTI provides multimodal inputs containing text and images and requires the models to give precise or open-ended answers, reflecting real-life exam styles. Additionally, MULTI includes over 18,000 questions, covering a variety of tasks from formula derivation to image detail analysis and cross-modal reasoning. The paper also introduces MULTI-ELITE (a subset containing 500 carefully selected challenging problems) and MULTI-EXTEND (a subset containing over 4,500 questions with external knowledge backgrounds) to further challenge the learning and knowledge transfer capabilities of MLLMs. Through these designs, MULTI not only provides a robust evaluation platform for MLLMs but also paves the way for the development of expert-level AI.

MULTI: Multimodal Understanding Leaderboard with Text and Images

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

MMR: Evaluating Reading Ability of Large Multimodal Models

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Published at NeurIPS 2024 Track Datasets and Benchmarks)

MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models

M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models

MileBench: Benchmarking MLLMs in Long Context

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

CMMLU: Measuring massive multitask language understanding in Chinese