Abstract:Neural sequence models based on the transformer architecture have demonstrated remarkable \emph{in-context learning} (ICL) abilities, where they can perform new tasks when prompted with training and test examples, without any parameter update to the model. This work first provides a comprehensive statistical theory for transformers to perform ICL. Concretely, we show that transformers can implement a broad class of standard machine learning algorithms in context, such as least squares, ridge regression, Lasso, learning generalized linear models, and gradient descent on two-layer neural networks, with near-optimal predictive power on various in-context data distributions. Using an efficient implementation of in-context gradient descent as the underlying mechanism, our transformer constructions admit mild size bounds, and can be learned with polynomially many pretraining sequences. Building on these ``base'' ICL algorithms, intriguingly, we show that transformers can implement more complex ICL procedures involving \emph{in-context algorithm selection}, akin to what a statistician can do in real life -- A \emph{single} transformer can adaptively select different base ICL algorithms -- or even perform qualitatively different tasks -- on different input sequences, without any explicit prompting of the right algorithm or task. We both establish this in theory by explicit constructions, and also observe this phenomenon experimentally. In theory, we construct two general mechanisms for algorithm selection with concrete examples: pre-ICL testing, and post-ICL validation. As an example, we use the post-ICL validation mechanism to construct a transformer that can perform nearly Bayes-optimal ICL on a challenging task -- noisy linear models with mixed noise levels. Experimentally, we demonstrate the strong in-context algorithm selection capabilities of standard transformer architectures.

Transformers are Deep Optimizers: Provable In-Context Learning for Deep Model Training

Transformers are Minimax Optimal Nonparametric In-Context Learners

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection

Pretrained transformer efficiently learns low-dimensional target functions in-context

In-context Learning on Function Classes Unveiled for Transformers

How Well Can Transformers Emulate In-context Newton's Method?

Transformers learn to implement preconditioned gradient descent for in-context learning

Transformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Models

In-Context Learning with Representations: Contextual Generalization of Trained Transformers

On the Training Convergence of Transformers for In-Context Classification

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression

Trained Transformers Learn Linear Models In-Context

Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens

How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?

Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions

Linear Transformers are Versatile In-Context Learners

Context-Scaling versus Task-Scaling in In-Context Learning

In-Context Convergence of Transformers

Provable In-Context Learning of Linear Systems and Linear Elliptic PDEs with Transformers

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning