GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers

Qintong Li,Leyang Cui,Xueliang Zhao,Lingpeng Kong,Wei Bi

2024-07-02

Abstract:Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. However, there are increasing debates regarding whether these models truly understand and apply mathematical knowledge or merely rely on shortcuts for mathematical reasoning. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations. We introduce the adversarial grade school math (GSM-Plus) dataset, an extension of GSM8K augmented with various mathematical perturbations. Our experiments on 25 LLMs and 4 prompting techniques show that while LLMs exhibit different levels of math reasoning abilities, their performances are far from robust. In particular, even for problems that have been solved in GSM8K, LLMs can make mistakes when new statements are added or the question targets are altered. We also explore whether more robust performance can be achieved by composing existing prompting methods, in which we try an iterative method that generates and verifies each intermediate thought based on its reasoning goal and calculation result.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the controversy between the true understanding capabilities and surface pattern recognition of large language models (LLMs) in mathematical reasoning. Specifically, although existing LLMs perform well on multiple mathematical benchmarks, it remains questionable whether these models truly understand and apply mathematical knowledge or merely rely on surface patterns for reasoning. To tackle this issue, the paper introduces an adversarial elementary math dataset called GSM-P LUS, which is based on the GSM8K dataset and includes various mathematical perturbations. Using this dataset, researchers tested 25 different LLMs and 4 prompting techniques to evaluate the robustness of these models when handling variations of mathematical problems. The experimental results show that although LLMs can achieve good performance on the original GSM8K dataset, their performance significantly drops when faced with variant problems in GSM-P LUS, especially in scenarios requiring critical thinking, arithmetic changes, and the insertion of distractors. Additionally, the paper explores a combined prompting method that enhances model robustness by iteratively generating and verifying each intermediate reasoning step. Overall, the paper calls for further research not only to improve the performance of LLMs on mathematical reasoning tasks but also to enhance their robustness in solving problems under different circumstances.

GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Investigating the Robustness of LLMs on Math Word Problems

MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments

MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

Multiple-Choice Questions are Efficient and Robust LLM Evaluators

Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist

Are Your LLMs Capable of Stable Reasoning?

Not All LLM Reasoners Are Created Equal

MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models

Cutting Through the Noise: Boosting LLM Performance on Math Word Problems

Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problems

Benchmarking Large Language Models for Math Reasoning Tasks

Common 7B Language Models Already Possess Strong Math Capabilities