Alec Radford,Christopher Hesse,Jeff Wu,Pranav Shyam,Gretchen Krueger,Sam McCandlish,T. Henighan,R. Child,S. Gray,Mark Chen,Amanda Askell,B. Chess,Prafulla Dhariwal,I. Sutskever,Eric Sigler,Arvind Neelakantan,Tom B. Brown,Ariel Herbert-Voss,Christopher Berner,Clemens Winter,A. Ramesh,Girish Sastry,Jack Clark,Benjamin Mann,Dario Amodei,Sandhini Agarwal,Nick Ryder,Daniel M. Ziegler,Melanie Subbiah,Ma-teusz Litwin,J. Kaplan

Abstract:Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Switch-GPT: An Effective Method for Constrained Text Generation under Few-Shot Settings (Student Abstract)

Few-shot Natural Language Generation for Task-Oriented Dialog

LOGEN: Few-shot Logical Knowledge-Conditioned Text Generation with Self-training.

Controllable Text Generation with Language Constraints

Controllable Text-to-Image Generation with GPT-4

Language Models are Few-Shot Learners

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding

TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching

Few-shot Learning with Multilingual Language Models

Why is constrained neural language generation particularly challenging?

Few-Shot Adaptation of Generative Adversarial Networks

NeuGPT: Unified multi-modal Neural GPT

AugGPT: Leveraging ChatGPT for Text Data Augmentation

Adapting Knowledge for Few-shot Table-to-Text Generation

ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles

Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Efficient and Training-Free Control of Language Generation

Switchable Decision: Dynamic Neural Generation Networks

A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation