Abstract:Mathematical reasoning presents a significant challenge to the cognitive capabilities of LLMs. Various methods have been proposed to enhance the mathematical ability of LLMs. However, few recognize the value of state transition for LLM reasoning. In this work, we define mathematical problem-solving as a process of transiting from an initial unsolved state to the final resolved state, and propose Kwai-STaR framework, which transforms LLMs into State-Transition Reasoners to improve their intuitive reasoning capabilities. Our approach comprises three main steps: (1) Define the state space tailored to the mathematical reasoning. (2) Generate state-transition data based on the state space. (3) Convert original LLMs into State-Transition Reasoners via a curricular training strategy. Our experiments validate the effectiveness of Kwai-STaR in enhancing mathematical reasoning: After training on the small-scale Kwai-STaR dataset, general LLMs, including Mistral-7B and LLaMA-3, achieve considerable performance gain on the GSM8K and GSM-Hard dataset. Additionally, the state transition-based design endows Kwai-STaR with remarkable training and inference efficiency. Further experiments are underway to establish the generality of Kwai-STaR.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the shortcomings of large language models (LLMs) in mathematical reasoning. Although various methods have been attempted to enhance the mathematical capabilities of LLMs, few have recognized the importance of state transitions in LLM reasoning. To this end, the authors propose a new framework—Kwai-STaR (State-Transition Reasoner), which improves the intuitive reasoning ability of LLMs by defining the process of solving mathematical problems as a series of state transitions from the initial unsolved problem state to the final solved problem state. ### Main Contributions 1. **New Perspective on State Transitions**: The authors propose a new perspective of modeling the mathematical reasoning process as state transitions and construct a state transition dataset. 2. **Kwai-STaR Framework**: This framework transforms general LLMs into state-transition reasoners (STaR) through state transitions, significantly improving their mathematical performance. 3. **Efficiency and Generalization**: Kwai-STaR not only performs well in terms of performance but also shows significant advantages in training and inference efficiency, demonstrating the great potential of state space strategies in enhancing LLM reasoning capabilities. ### Experimental Results Experiments show that the Kwai-STaR framework significantly improves the performance of multiple general LLMs in the GSM8K and GSM-Hard benchmarks. Compared to existing data augmentation methods, Kwai-STaR achieves greater performance improvements with smaller data scales and fewer trainable parameters. Additionally, the accuracy of Kwai-STaR in single inference can rival the multiple inference accuracy of other multi-step reasoning methods, without the need for complex reasoning paradigms and high inference costs. ### Future Work The authors plan to further validate the feasibility of Kwai-STaR in other domains and provide more diverse experimental results to demonstrate its generalization capabilities. Moreover, they hope to improve the design of the state space to make it more complete and automated, and explore theoretical explanations of how state space enhances LLM reasoning capabilities.

Kwai-STaR: Transform LLMs into State-Transition Reasoners

Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model

MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time

KwaiYiiMath: Technical Report

Enhancing Mathematical Reasoning in LLMs by Stepwise Correction

Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo

Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers

Specialized Mathematical Solving by a Step-By-Step Expression Chain Generation

Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning

Learning Multi-Step Reasoning by Solving Arithmetic Tasks

Logic Contrastive Reasoning with Lightweight Large Language Model for Math Word Problems

Do Large Language Models Truly Grasp Mathematics? An Empirical Exploration From A Psychological Perspective

Multi-tool Integration Application for Math Reasoning Using Large Language Model

Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation

S$^3$c-Math: Spontaneous Step-level Self-correction Makes Large Language Models Better Mathematical Reasoners

CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning

Enhancing Mathematical Reasoning in LLMs with Background Operators

Arithmetic Reasoning with LLM: Prolog Generation & Permutation

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning