Alec Radford,Christopher Hesse,Jeff Wu,Pranav Shyam,Gretchen Krueger,Sam McCandlish,T. Henighan,R. Child,S. Gray,Mark Chen,Amanda Askell,B. Chess,Prafulla Dhariwal,I. Sutskever,Eric Sigler,Arvind Neelakantan,Tom B. Brown,Ariel Herbert-Voss,Christopher Berner,Clemens Winter,A. Ramesh,Girish Sastry,Jack Clark,Benjamin Mann,Dario Amodei,Sandhini Agarwal,Nick Ryder,Daniel M. Ziegler,Melanie Subbiah,Ma-teusz Litwin,J. Kaplan

Abstract:Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Out of One, Many: Using Language Models to Simulate Human Samples

Artificial intelligence in psychology research

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information

Diminished diversity-of-thought in a standard large language model

From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models

Towards Coding Social Science Datasets with Language Models

Generative Language Models Exhibit Social Identity Biases

ChatGPT vs Social Surveys: Probing the Objective and Subjective Human Society

Perils and opportunities in using large language models in psychological research

Can AI language models replace human participants?

Human Simulacra: Benchmarking the Personification of Large Language Models

Large language models as linguistic simulators and cognitive models in human research

Large Language Models Can Infer Psychological Dispositions of Social Media Users

Can AI Replace Human Subjects? A Large-Scale Replication of Psychological Experiments with LLMs

I Am Not Them: Fluid Identities and Persistent Out-group Bias in Large Language Models

Large Language Models as Subpopulation Representative Models: A Review

Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoption

Language Models are Few-Shot Learners

Exposing Bias in Online Communities through Large-Scale Language Models

Exploring the Application Potential of the Large Language Model in Sociological Research: A Case Study of ChatGPT