Abstract:Large transformer-based models are able to perform in-context few-shot learning, without being explicitly trained for it. This observation raises the question: what aspects of the training regime lead to this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself. In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having large numbers of rarely occurring classes. In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed. These properties are exemplified by natural language, but are also inherent to naturalistic data in a wide range of other domains. They also depart significantly from the uniform, i.i.d. training distributions typically used for standard supervised learning. In our initial experiments, we found that in-context learning traded off against more conventional weight-based learning, and models were unable to achieve both simultaneously. However, our later experiments uncovered that the two modes of learning could co-exist in a single model when it was trained on data following a skewed Zipfian distribution -- another common property of naturalistic data, including language. In further experiments, we found that naturalistic data distributions were only able to elicit in-context learning in transformers, and not in recurrent models. In sum, our findings indicate how the transformer architecture works together with particular properties of the training data to drive the intriguing emergent in-context learning behaviour of large language models, and how future work might encourage both in-context and in-weights learning in domains beyond language.

The Developmental Landscape of In-Context Learning

Breaking through the learning plateaus of in-context learning in Transformer

Toward Understanding In-context vs. In-weight Learning

A Theory of Emergent In-Context Learning as Implicit Structure Induction

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions

In-Context Learning with Representations: Contextual Generalization of Trained Transformers

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data

In-Context Convergence of Transformers

Enhanced Transformer architecture for in-context learning of dynamical systems

Data Distributional Properties Drive Emergent In-Context Learning in Transformers

MLPs Learn In-Context on Regression and Classification Tasks

Does learning the right latent variables necessarily improve in-context learning?

Asymptotic theory of in-context learning by linear attention

Do pretrained Transformers Learn In-Context by Gradient Descent?

In-context Learning in Presence of Spurious Correlations

On the Role of Depth and Looping for In-Context Learning with Task Diversity

What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning

Generalization to New Sequential Decision Making Tasks with In-Context Learning

Context-Scaling versus Task-Scaling in In-Context Learning