Abstract:Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available.

What problem does this paper attempt to address?

The problem this paper attempts to address is the current lack of detailed and diverse benchmark tools for evaluating the mathematical abilities of large multimodal models in non-English environments, particularly in Chinese contexts. Although datasets like MathVista have been proposed to assess mathematical abilities, these datasets mainly focus on English environments and lack tools for fine-grained evaluation of Chinese multimodal math problems in the K12 educational context. To fill this gap, the authors propose a new Chinese multimodal math skill evaluation benchmark—CMMaTH (Chinese Multi-modal Math Skill Evaluation Benchmark). CMMaTH contains 23,000 multimodal problems related to K12 mathematics, making it the largest Chinese multimodal math problem benchmark dataset to date. These problems range from elementary to high school levels, providing various types of problems, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. Additionally, the authors developed an open-source tool called GradeGPT to compare the consistency of different model outputs with standard answers, enabling stable, fast, and free model evaluation. Using the CMMaTH dataset and the GradeGPT tool, the authors systematically evaluated mainstream open-source and commercial large multimodal models, reporting comprehensive evaluation results and extensive case analyses. In summary, the main contributions of this paper include: 1. Proposing the most detailed and high-quality Chinese multimodal math benchmark dataset to date, CMMaTH, and providing an English version of this dataset. 2. Demonstrating greater reasoning depth and diversity compared to existing multimodal math benchmarks, simulating more realistic educational Q&A scenarios, and covering a wider range of problem types and answer formats. 3. Developing an evaluation assistant named GradeGPT, which can compare model responses with standard answers to assess the correctness of results and processes. 4. Systematically evaluating existing mainstream large multimodal models, quantitatively and qualitatively comparing their performance.

CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models

CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models

MathScape: Evaluating MLLMs in multimodal Math Scenarios through a Hierarchical Benchmark

MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models

Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models

Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning

InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning

MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model

MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation

GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving

MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions

SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese