Abstract:Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction (a.k.a. conformal inference) is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, timeseries, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run by following the code footnotes.

Does confidence calibration improve conformal prediction?

Delving into Temperature Scaling for Adaptive Conformal Prediction

On Temperature Scaling and Conformal Prediction of Deep Classifiers

Conformal Prediction Sets Improve Human Decision Making

Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage

Conformal Prediction for Deep Classifier via Label Ranking

Improving Expert Predictions with Conformal Prediction

Trustworthy Classification through Rank-Based Conformal Prediction Sets

Conformalized Interval Arithmetic with Symmetric Calibration

Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration

Improving Adaptive Conformal Prediction Using Self-Supervised Learning

Self-Calibrating Conformal Prediction

Rethinking Confidence Calibration for Failure Prediction

Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors

Nothing Conformal about Adaptive Conformal Inference

Entropy Reweighted Conformal Classification

Confidence Calibration for Systems with Cascaded Predictive Modules

Adaptive Conformal Inference by Betting

Conformal Prediction: A Gentle Introduction

C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets

Boosted Conformal Prediction Intervals