Abstract:This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like reasoning actions to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered mutual consistent, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral-7B, from 74.53% to 91.13% for LLaMA3-8B-Instruct. Code will be available at <a class="link-external link-https" href="https://github.com/zhentingqi/rStar" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper primarily addresses the limitations of Small Language Models (SLMs) in complex reasoning tasks and proposes a new method called rStar, aimed at significantly enhancing the reasoning capabilities of these models. Specifically, rStar tackles the problem through the following two key steps: 1. **Self-play Mutual Reasoning**: This method decomposes the reasoning process into two parts: Generation and Discrimination. First, the target SLM generates high-quality reasoning trajectories through enhanced Monte Carlo Tree Search (MCTS). Then, another SLM with similar capabilities acts as a discriminator to verify these trajectories generated by the target SLM. If both SLMs agree on the same trajectory, it is considered mutually consistent and thus more likely to be correct. 2. **Rich Set of Reasoning Actions**: To improve the effectiveness of exploring the solution space, rStar introduces a rich set of human-like reasoning actions, including proposing a single step of thought, proposing the remaining steps of thought, proposing the next sub-question and its answer, re-answering sub-questions, and restating questions/sub-questions, among others. These actions enable the SLM to better generate high-quality candidate reasoning trajectories. Experimental results show that rStar significantly improves the reasoning performance of SLMs on multiple benchmark datasets. For example, on the GSM8K dataset, rStar greatly enhances the accuracy of different scales of SLMs (such as LLaMA2-7B, Mistral-7B, and LLaMA3-8B), even matching or surpassing models that have been specifically fine-tuned. In summary, the paper addresses the two major challenges faced by existing SLMs in complex reasoning tasks: the difficulty in effectively exploring the solution space and the difficulty in accurately evaluating the quality of reasoning steps. By introducing the self-play mutual reasoning method and a rich set of reasoning actions, rStar successfully enhances the reasoning capabilities of SLMs.

Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers

Learning to Reason via Self-Iterative Process Feedback for Small Language Models

MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding

Interpretable Contrastive Monte Carlo Tree Search Reasoning

Can LLMs Reason in the Wild with Programs?

LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

Kwai-STaR: Transform LLMs into State-Transition Reasoners

Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo

LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning

SMART: Self-learning Meta-strategy Agent for Reasoning Tasks

Democratizing Reasoning Ability: Tailored Learning from Large Language Model

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards

V-STaR: Training Verifiers for Self-Taught Reasoners

RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner

LLMs for Relational Reasoning: How Far are We?