Abstract:This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the shortcomings of large language models (LLMs) in mathematical reasoning and problem-solving abilities. Despite their excellent performance in many areas, LLMs still have limited capabilities when dealing with complex mathematical problems. Specifically, the paper focuses on how to enhance the mathematical reasoning and problem-solving abilities of LLMs by combining Chain-of-Thought (CoT) and Program-of-Thought (PoT) learning methods. ### Background and Motivation 1. **Importance of Mathematical Reasoning**: Mathematical reasoning involves not only numerical calculations but also logical thinking, problem-solving, and complex decision-making abilities, which are crucial for understanding and generating human-like responses. 2. **Existing Challenges**: Current LLMs have not yet reached the level of their language processing capabilities in mathematical reasoning, limiting their applications in fields requiring numerical and logical understanding, such as programming. 3. **Limitations of Existing Methods**: - **CoT Learning**: While it can improve reasoning abilities, it is prone to arithmetic errors when dealing with large numbers. - **PoT Learning**: By expressing reasoning steps as code and delegating them to a code interpreter, it can improve computational accuracy but requires the reasoning steps to be accurately expressed as code. ### Research Objectives 1. **Propose a New Learning Strategy**: The paper proposes a sequential learning strategy called SAAS (Solving Ability Amplification Strategy), which transitions from CoT learning to PoT learning to enhance problem-solving abilities. 2. **Validate the Strategy's Effectiveness**: Through extensive experiments, validate the performance of SAAS across multiple benchmarks, demonstrating its superiority in mathematical reasoning and problem-solving abilities. ### Main Contributions 1. **Sequential Learning Strategy**: SAAS ensures that the model first undergoes CoT learning and then PoT learning, thereby enhancing problem-solving abilities on the foundation of logical reasoning skills. 2. **Cognitive Retention Strategy**: During the PoT learning phase, random sampling of CoT data samples is used to prevent cognitive forgetting, ensuring the model retains previously learned reasoning skills. 3. **Empirical Results**: Experimental results show that SAAS achieves state-of-the-art performance across multiple benchmarks, significantly outperforming existing competitors. ### Conclusion By proposing the SAAS strategy, the paper successfully addresses the shortcomings of LLMs in mathematical reasoning and problem-solving abilities, providing new perspectives and methods for research in this field. However, the study also highlights the limitations of LLMs in mathematical reasoning, including the unpredictability of outputs, insufficient understanding of advanced mathematical concepts, reliance on synthetic data, and ethical considerations, emphasizing the directions and challenges for future research.

SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models

SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models

Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning

Achieving >97 Better Solvers for Math Word Problems

Specialized Mathematical Solving by a Step-By-Step Expression Chain Generation

Learning Multi-Step Reasoning by Solving Arithmetic Tasks

Solving Math Word Problems by Combining Language Models With Symbolic Solvers

Expanding Search Space with Diverse Prompting Agents: An Efficient Sampling Approach for LLM Mathematical Reasoning

MathLearner: A Large Language Model Agent Framework for Learning to Solve Mathematical Problems

Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model

Boosting Large Language Models with Socratic Method for Conversational Mathematics Teaching

LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning

MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning

BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning

Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

Stepwise Self-Consistent Mathematical Reasoning with Large Language Models

StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving

Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models

Strategic Chain-of-Thought: Guiding Accurate Reasoning in LLMs through Strategy Elicitation

Enhancing Mathematical Reasoning in LLMs by Stepwise Correction