Abstract:Despite the existence of various benchmarks for evaluating natural language processing models, we argue that human exams are a more suitable means of evaluating general intelligence for large language models (LLMs), as they inherently demand a much wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. To this end, we introduce M3Exam, a novel benchmark sourced from real and official human exam questions for evaluating LLMs in a multilingual, multimodal, and multilevel context. M3Exam exhibits three unique characteristics: (1) multilingualism, encompassing questions from multiple countries that require strong multilingual proficiency and cultural knowledge; (2) multimodality, accounting for the multimodal nature of many exam questions to test the model's multimodal understanding capability; and (3) multilevel structure, featuring exams from three critical educational periods to comprehensively assess a model's proficiency at different levels. In total, M3Exam contains 12,317 questions in 9 diverse languages with three educational levels, where about 23\% of the questions require processing images for successful solving. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text, particularly in low-resource and non-Latin script languages. Multimodal LLMs also perform poorly with complex multimodal questions. We believe that M3Exam can be a valuable resource for comprehensively evaluating LLMs by examining their multilingual and multimodal abilities and tracking their development. Data and evaluation code is available at \url{https://github.com/DAMO-NLP-SG/M3Exam}.

What problem does this paper attempt to address?

The paper attempts to address the limitations of existing benchmark methods for evaluating large language models (LLMs), which include: 1. **Insufficient Multilingual Capability Evaluation**: Most existing benchmarks primarily focus on English, neglecting the evaluation of models in multilingual environments. Many LLMs possess multilingual capabilities and are widely used in different countries and languages, making it crucial to assess their multilingual performance. 2. **Insufficient Multimodal Capability Evaluation**: Existing benchmarks mostly consider text-based questions, ignoring that real-world exam questions often include multimodal information such as images. Evaluating the model's ability to handle multimodal information is essential for practical applications. 3. **Insufficient Multi-level Evaluation**: Current exam-type benchmarks typically draw questions from mixed exams, such as university final exams or professional certification tests, leading to resources containing questions of different levels. Collecting exam questions from different educational stages is important for evaluating and understanding LLMs' performance at various intellectual levels. To address these issues, the authors propose a new benchmark dataset called M3Exam, which has the following features: - **Multilinguality**: Collects naturally multilingual questions from official exams in multiple countries, preserving the socio-cultural diversity of different languages. - **Multimodality**: Includes questions that require images and processes these images to facilitate model evaluation. Approximately 23% of the questions require image processing to be successfully answered. - **Multi-level Structure**: Collects official exam questions from three key educational stages (primary school, middle school, and high school), forming a multi-tiered benchmark dataset. Through M3Exam, the authors aim to provide a resource for comprehensively evaluating LLMs' performance in multilingual and multimodal settings and tracking their development.

M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading

Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

A Survey on Benchmarks of Multimodal Large Language Models

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

OMGEval: an Open Multilingual Generative Evaluation Benchmark for Large Language Models

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmark

M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models

LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models

MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs

MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models

MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models

SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

CJEval: A Benchmark for Assessing Large Language Models Using Chinese Junior High School Exam Data