Abstract:State-of-the-art large language models (LLMs) exhibit impressive problem-solving capabilities but may struggle with complex reasoning and factual correctness. Existing methods harness the strengths of chain-of-thought and retrieval-augmented generation (RAG) to decompose a complex problem into simpler steps and apply retrieval to improve factual correctness. These methods work well on straightforward reasoning tasks but often falter on challenging tasks such as competitive programming and mathematics, due to frequent reasoning errors and irrelevant knowledge retrieval. To address this, we introduce Critic-guided planning with Retrieval-augmentation, CR-Planner, a novel framework that leverages fine-tuned critic models to guide both reasoning and retrieval processes through planning. CR-Planner solves a problem by iteratively selecting and executing sub-goals. Initially, it identifies the most promising sub-goal from reasoning, query generation, and retrieval, guided by rewards given by a critic model named sub-goal critic. It then executes this sub-goal through sampling and selecting the optimal output based on evaluations from another critic model named execution critic. This iterative process, informed by retrieved information and critic models, enables CR-Planner to effectively navigate the solution space towards the final answer. We employ Monte Carlo Tree Search to collect the data for training the critic models, allowing for a systematic exploration of action sequences and their long-term impacts. We validate CR-Planner on challenging domain-knowledge-intensive and reasoning-heavy tasks, including competitive programming, theorem-driven math reasoning, and complex domain retrieval problems. Our experiments demonstrate that CR-Planner significantly outperforms baselines, highlighting its effectiveness in addressing challenging problems by improving both reasoning and retrieval.

Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search

Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

Improve Mathematical Reasoning in Language Models by Automated Process Supervision

AlphaMath Almost Zero: Process Supervision without Process

No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function

Interpretable Contrastive Monte Carlo Tree Search Reasoning

SRA-MCTS: Self-driven Reasoning Augmentation with Monte Carlo Tree Search for Code Generation

Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning

Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning

Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo

When Do Program-of-Thought Works for Reasoning?

Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design

Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuning

Learning to Reason via Self-Iterative Process Feedback for Small Language Models

Technical Report: Enhancing LLM Reasoning with Reward-guided Tree Search

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data

Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers

Progressive Multimodal Reasoning via Active Retrieval

Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B