Small Language Models Need Strong Verifiers to Self-Correct Reasoning

Yunxiang Zhang,Muhammad Khalifa,Lajanugen Logeswaran,Jaekyeom Kim,Moontae Lee,Honglak Lee,Lu Wang

2024-06-06

Abstract:Self-correction has emerged as a promising solution to boost the reasoning performance of large language models (LLMs), where LLMs refine their solutions using self-generated critiques that pinpoint the errors. This work explores whether small (<= 13B) language models (LMs) have the ability of self-correction on reasoning tasks with minimal inputs from stronger LMs. We propose a novel pipeline that prompts smaller LMs to collect self-correction data that supports the training of self-refinement abilities. First, we leverage correct solutions to guide the model in critiquing their incorrect responses. Second, the generated critiques, after filtering, are used for supervised fine-tuning of the self-correcting reasoner through solution refinement. Our experimental results show improved self-correction abilities of two models on five datasets spanning math and commonsense reasoning, with notable performance gains when paired with a strong GPT-4-based verifier, though limitations are identified when using a weak self-verifier for determining when to correct.

Computation and Language

What problem does this paper attempt to address?

This paper discusses the self-correction ability of small-scale language models (LLMs) in inference tasks. Typically, large-scale language models can identify and correct their incorrect answers by generating critiques. However, small-scale models have weaker self-correction abilities and rely on guidance from large-scale models. The study proposes a new method called SCORE, which enables small-scale models to collect self-correction data and enhance their self-correction ability through supervised fine-tuning.

Specifically, SCORE first uses correct answers to guide the model to critique its incorrect responses, and then filters and uses these critiques for supervised fine-tuning to train a self-correction reasoning system. The experiments show that this method improves the self-correction ability of both models on five datasets, especially when combined with the powerful GPT-4 baseline validator. However, performance decreases when a weak self-validator is used to determine when to make corrections.

The paper also points out that while small-scale models can learn self-correction through distillation from powerful models, this approach poses security risks and hinders scientific understanding of enhancing models' error correction abilities. The SCORE method avoids supervision from powerful models, enabling small-scale models to self-improve their correction abilities. The research results demonstrate that, even with limited data, small-scale models can significantly enhance their self-correction abilities through the SCORE method.