Abstract:There has been considerable divergence of opinion on the reasoning abilities of Large Language Models (LLMs). While the initial optimism that reasoning might emerge automatically with scale has been tempered thanks to a slew of counterexamples--ranging from multiplication to simple planning--there persists a wide spread belief that LLMs can self-critique and improve their own solutions in an iterative fashion. This belief seemingly rests on the assumption that verification of correctness should be easier than generation--a rather classical argument from computational complexity--which should be irrelevant to LLMs to the extent that what they are doing is approximate retrieval. In this paper, we set out to systematically investigate the effectiveness of iterative prompting in the context of reasoning and planning. We present a principled empirical study of the performance of GPT-4 in three domains: Game of 24, Graph Coloring, and STRIPS planning. We experiment both with the model critiquing its own answers and with an external correct reasoner verifying proposed solutions. In each case, we analyze whether the content of criticisms actually affects bottom line performance, and whether we can ablate elements of the augmented system without losing performance. We observe significant performance collapse with self-critique and significant performance gains with sound external verification. We also note that merely re-prompting with a sound verifier maintains most of the benefits of more involved setups.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to explore the self-verification capabilities of large language models (LLMs) in reasoning and planning tasks. Despite initial optimism that reasoning abilities might emerge automatically with scale, a series of counterexamples indicate that LLMs do not perform well on certain simple tasks (such as multiplication or simple planning). Nevertheless, there is a prevailing view that LLMs can iteratively self-criticize and improve their solutions. This view is based on an assumption: verifying correctness should be easier than generating answers, a classic argument in computational complexity theory. However, the authors are skeptical of this assumption, suggesting that it may not hold for LLMs, as their behavior resembles approximate retrieval. To systematically study the effectiveness of iterative prompting in reasoning and planning tasks, the authors selected three specific tasks: the 24-point game, the graph coloring problem, and STRIPS planning. They experimentally tested the following scenarios: 1. The model self-criticizes its answers. 2. An external correct reasoner verifies the proposed solutions. The study results indicate that performance significantly declines when the model self-criticizes, whereas performance significantly improves when using an external correct verifier. Additionally, even simply re-prompting with the correct verifier maintains most of the improvement. ### Key Findings - **Limitations of Self-Criticism**: With an increase in self-criticism iterations, the output quality continuously declines, primarily due to the high false negative rate of the verifier LLM, meaning that even if the generated answers are correct, the verifier often incorrectly rejects them. - **Importance of External Verification**: Using an external correct verifier can significantly improve performance, and the improvement persists even with minimal or no feedback provided. - **Effectiveness of Simplified Systems**: Even completely removing criticism and merely repeating the base prompt query can retain most of the performance gains brought by criticism. ### Conclusion The authors suggest that future implementations should adopt an LLM-Modulo system, where verification is performed by an external correct system rather than relying on the LLM's self-criticism. This helps avoid performance degradation due to self-criticism and ensures higher accuracy.

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

Can Large Language Models Really Improve by Self-critiquing Their Own Plans?

Large Language Models Are Better Reasoners with Self-Verification

GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems

Small Language Models Need Strong Verifiers to Self-Correct Reasoning

General Purpose Verification for Chain of Thought Prompting

A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning

Large Language Models are reasoners with Self-Verification

On the Planning Abilities of Large Language Models : A Critical Investigation

Reasoning with Large Language Models, a Survey

Can Large Language Models Reason and Plan?

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

Large Language Models Cannot Self-Correct Reasoning Yet

Deductive Verification of Chain-of-Thought Reasoning

GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach

Can Large Language Models Reason? A Characterization via 3-SAT

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)

LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks

Large Language Models Can Self-Correct with Key Condition Verification

Case Study: Testing Model Capabilities in Some Reasoning Tasks