Abstract:Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics. To address these concerns, we conduct a large-scale study on several SOTA open and closed models. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of <a class="link-external link-http" href="http://models.Our" rel="external noopener nofollow">this http URL</a> findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and show that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is because current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data. Adding a single clause that seems relevant to the question causes significant performance drops (up to 65%) across all state-of-the-art models, even though the clause doesn't contribute to the reasoning chain needed for the final answer. Overall, our work offers a more nuanced understanding of LLMs' capabilities and limitations in mathematical reasoning.

Over-Reasoning and Redundant Calculation of Large Language Models

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Analysis of the Reasoning with Redundant Information Provided Ability of Large Language Models

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Rational Metareasoning for Large Language Models

Evaluating Consistency and Reasoning Capabilities of Large Language Models

On Memorization of Large Language Models in Logical Reasoning

Interpreting and Improving Large Language Models in Arithmetic Calculation

Logic Contrastive Reasoning with Lightweight Large Language Model for Math Word Problems

CLR-Bench: Evaluating Large Language Models in College-level Reasoning

Concise and Organized Perception Facilitates Reasoning in Large Language Models

Do Large Language Models Truly Grasp Mathematics? An Empirical Exploration From A Psychological Perspective

Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model

NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models

Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

Arithmetic Reasoning with LLM: Prolog Generation & Permutation

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification