Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

Eric Zelikman,Georges Harik,Yijia Shao,Varuna Jayasiri,Nick Haber,Noah D. Goodman
2024-03-18
Abstract:When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%$\rightarrow$10.9%) and CommonsenseQA (36.3%$\rightarrow$47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The problem this paper attempts to address is how to enable language models (LMs) to self-reason while generating text and learn from it to improve their ability to predict future text. Specifically, the paper proposes the Quiet-STaR method, a new technique for inferring unstated reasoning from arbitrary text. Unlike previous methods that were limited to specific tasks or datasets, Quiet-STaR aims to enable language models to interpret future text content by automatically generating reasoning when processing internet text corpora, thereby enhancing prediction accuracy. The authors found that existing reasoning enhancement methods often rely on carefully curated datasets, which limits their generalization ability and scale. Therefore, they proposed a new approach that allows language models to learn reasoning while handling various natural language tasks, rather than just specific tasks. Through this method, language models can achieve significant zero-shot learning improvements on tasks such as CommonsenseQA and GSM8K without fine-tuning. Additionally, the paper explores how generating reasoning helps models predict difficult vocabulary and demonstrates how these improvements continue to increase with the number of internal thinking words.