Abstract:Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction (a.k.a. conformal inference) is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, timeseries, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run by following the code footnotes.

Conformal Prediction with Partially Labeled Data

Conformal Prediction with Learned Features

Conformal Prediction for Deep Classifier via Label Ranking

Conformal Prediction: A Gentle Introduction

Split Conformal Prediction under Data Contamination

Conformalization of Sparse Generalized Linear Models

Conformal prediction for robust deep nonparametric regression

Self-Calibrating Conformal Prediction

A conformalized learning of a prediction set with applications to medical imaging classification

Probabilistic Conformal Prediction with Approximate Conditional Validity

Robust Yet Efficient Conformal Prediction Sets

Making learning more transparent using conformalized performance prediction

Stochastic Online Conformal Prediction with Semi-Bandit Feedback

Improving Expert Predictions with Conformal Prediction

Conformal Prediction via Regression-as-Classification

Conformalized Credal Set Predictors

A Conformal Prediction Score that is Robust to Label Noise

Predictive Inference with Feature Conformal Prediction

Distribution-Free Finite-Sample Guarantees and Split Conformal Prediction

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Conformal Prediction Sets Improve Human Decision Making