Abstract:When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%$\rightarrow$10.9%) and CommonsenseQA (36.3%$\rightarrow$47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.

V-STaR: Training Verifiers for Self-Taught Reasoners

Lean-STaR: Learning to Interleave Thinking and Proving

RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

Small Language Models Need Strong Verifiers to Self-Correct Reasoning

Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors

Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

Large Language Models are reasoners with Self-Verification

Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers

Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo

Generative Verifiers: Reward Modeling as Next-Token Prediction

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

Token-Supervised Value Models for Enhancing Mathematical Reasoning Capabilities of Large Language Models

Rationale-Aware Answer Verification by Pairwise Self-Evaluation

Learning to Reason via Self-Iterative Process Feedback for Small Language Models

GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach

VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency

Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning

Prover-Verifier Games improve legibility of LLM outputs