Abstract:We study the task of prompting large-scale language models to perform multi-step reasoning. Existing work shows that when prompted with a chain of thoughts (CoT), sequences of short sentences describing intermediate reasoning steps towards a final answer, large language models can generate new reasoning chains and predict answers for new inputs. A central question is which reasoning examples make the most effective prompts. In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning. We show that prompts with higher reasoning complexity, i.e., chains with more reasoning steps, achieve substantially better performance on multi-step reasoning tasks over strong baselines. We further extend our complexity-based criteria from prompting (selecting inputs) to decoding (selecting outputs), where we sample multiple reasoning chains from the model, then choose the majority of generated answers from complex reasoning chains (over simple chains). When used to prompt GPT-3 and Codex, our approach substantially improves multi-step reasoning accuracy and achieves new state-of-the-art (SOTA) performance on three math benchmarks (GSM8K, MultiArith, and MathQA) and two BigBenchHard tasks (Date Understanding and Penguins), with an average +5.3 and up to +18 accuracy improvements. Compared with existing example selection schemes like manual tuning or retrieval-based selection, selection based on reasoning complexity is intuitive, easy to implement, and annotation-efficient. Further results demonstrate the robustness of performance gains from complex prompts under format perturbation and distribution shift.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation

Why Can Large Language Models Generate Correct Chain-of-Thoughts?

Large Language Models as Analogical Reasoners

Chain-of-Thought Reasoning Without Prompting

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Automatic Chain of Thought Prompting in Large Language Models

Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models

Reasoning with Large Language Models, a Survey

Pattern-Aware Chain-of-Thought Prompting in Large Language Models

What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

Complexity-Based Prompting for Multi-Step Reasoning

Chain-of-Thought Augmentation with Logit Contrast for Enhanced Reasoning in Language Models

Boosting Language Models Reasoning with Chain-of-Knowledge Prompting

An automatically discovered chain-of-thought prompt generalizes to novel models and datasets

Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models

Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance