Abstract:Recent advancements in Large Language Models (LLMs) have showcased striking results on existing logical reasoning benchmarks, with some models even surpassing human performance. However, the true depth of their competencies and robustness in reasoning tasks remains an open question. To this end, in this paper, we focus on two popular reasoning tasks: arithmetic reasoning and code generation. Particularly, we introduce (i) a general ontology of perturbations for math and coding questions, (ii) a semi-automatic method to apply these perturbations, and (iii) two datasets, GSMORE and HUMANEVAL-CORE, respectively, of perturbed math and coding problems to probe LLM capabilities in numeric reasoning and coding tasks. Through comprehensive evaluations of both closed-source and open-source LLMs, we show a significant performance drop across all the models against the perturbed questions, suggesting that the current LLMs lack robust problem solving skills and structured reasoning abilities in many areas, as defined by our ontology. We open-source the datasets and source codes at: <a class="link-external link-https" href="https://github.com/declare-lab/LLM-ReasoningTest" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem this paper attempts to address is the evaluation of large language models (LLMs) in their capabilities and robustness in mathematical reasoning and code generation tasks. Specifically, the authors focus on the following two aspects: 1. **Mathematical Reasoning Ability**: Although current LLMs' performance on mathematical problems is impressive, their ability to handle complex logical reasoning and structured problems still needs thorough evaluation. The paper introduces a series of perturbation operations to test LLMs' performance when faced with minor changes, aiming to reveal whether these models truly understand the underlying concepts and structures. 2. **Code Generation Ability**: Similarly, LLMs' performance in code generation tasks faces similar issues. The authors aim to evaluate these models' robustness and flexibility in handling code generation tasks by introducing perturbation operations. To achieve this goal, the authors propose a general perturbation ontology and develop a semi-automated method to apply these perturbations. They also created two datasets: GSM ORE and HUMAN EVAL-CORE, for mathematical reasoning and code generation tasks, respectively. Through these datasets, the authors conducted a comprehensive evaluation of multiple closed-source and open-source LLMs. The results show that all models exhibit significant performance degradation when faced with perturbed problems, indicating that current LLMs still lack sufficient reasoning ability and robustness in many areas. In summary, this paper aims to deeply evaluate and understand LLMs' true capabilities in mathematical reasoning and code generation tasks by systematically introducing perturbation operations, thereby providing directions for future model improvements.

Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering

An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning

Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving

MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data

LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs

Competition-Level Problems are Effective LLM Evaluators

Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

INC-Math: Integrating Natural Language and Code for Enhanced Mathematical Reasoning in Large Language Models

Large Language Models for Mathematical Reasoning: Progresses and Challenges

InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

Do Large Language Models Truly Grasp Mathematics? An Empirical Exploration From A Psychological Perspective

DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents

Can LLMs Reason in the Wild with Programs?

Evaluating Mathematical Reasoning Beyond Accuracy

Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification