Abstract:Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction (a.k.a. conformal inference) is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, timeseries, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run by following the code footnotes.

Transductive conformal inference with adaptive scores

Asymptotics for conformal inference

Adaptive inference with random ellipsoids through Conformal Conditional Linear Expectation

Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data

Adaptive Conformal Regression with Jackknife+ Rescaled Scores

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Nothing Conformal about Adaptive Conformal Inference

Selection by Prediction with Conformal p-values

Integrative conformal p -values for out-of-distribution testing with labelled outliers

Conformal Inference for Invariant Risk Minimization

Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)

Conformal Prediction Under Covariate Shift

Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage

Model-free selective inference under covariate shift via weighted conformal p-values

Probabilistic Conformal Prediction with Approximate Conditional Validity

Classification with Valid and Adaptive Coverage

Multi-Source Conformal Inference Under Distribution Shift

Inductive Conformal Prediction under Data Scarcity: Exploring the Impacts of Nonconformity Measures

Conformal Prediction: A Gentle Introduction

Predictive Inference with Feature Conformal Prediction

The Penalized Inverse Probability Measure for Conformal Classification