Class-Conditional Conformal Prediction with Many Classes

Tiffany Ding,Anastasios N. Angelopoulos,Stephen Bates,Michael I. Jordan,Ryan J. Tibshirani

2023-10-28

Abstract:Standard conformal prediction methods provide a marginal coverage guarantee, which means that for a random test point, the conformal prediction set contains the true label with a user-specified probability. In many classification problems, we would like to obtain a stronger guarantee--that for test points of a specific class, the prediction set contains the true label with the same user-chosen probability. For the latter goal, existing conformal prediction methods do not work well when there is a limited amount of labeled data per class, as is often the case in real applications where the number of classes is large. We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores and performs conformal prediction at the cluster level. Based on empirical evaluation across four image data sets with many (up to 1000) classes, we find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.

Machine Learning,Computer Vision and Pattern Recognition,Methodology

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in multi - class classification tasks, the existing Standard Conformal Prediction (SCP) cannot provide sufficient class - conditional coverage. Specifically, SCP can ensure that the true label is included with a probability specified by the user as a whole. However, when there are a large number of classes in the dataset and the labeled data for each class is limited, for a test point of a specific class, the prediction set may not be able to include the true label with the same probability. This deficiency may lead to poor prediction performance for some important classes. Especially in application scenarios such as medical diagnosis, it may cause patients to miss critical treatment. The paper proposes a new method - Clustered Conformal Prediction (CCP). By clustering classes with similar conformal score distributions together and performing conformal prediction at the cluster level, it improves class - conditional coverage. Experimental results show that CCP outperforms the existing SCP and Classwise Conformal Prediction (CCPW) methods on multiple image datasets. In particular, when the number of classes is large and the amount of data for each class is limited, CCP has better performance in terms of class - conditional coverage and prediction set size.

Class-Conditional Conformal Prediction with Many Classes

Conformal prediction with conditional guarantees

Posterior Conformal Prediction

Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage

Conformal Prediction with Learned Features

Probabilistic Conformal Prediction with Approximate Conditional Validity

A conformalized learning of a prediction set with applications to medical imaging classification

Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration

Trustworthy Classification through Rank-Based Conformal Prediction Sets

Conformal Prediction for Deep Classifier via Label Ranking

Aggregating Predictions on Multiple Non-disclosed Datasets using Conformal Prediction

Predictive Inference with Feature Conformal Prediction

Conformal Prediction using Conditional Histograms

Conformal Prediction via Regression-as-Classification

Weighted Aggregation of Conformity Scores for Classification

Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

Conformal Structured Prediction

Selecting informative conformal prediction sets with false coverage rate control

Multi-model Ensemble Conformal Prediction in Dynamic Environments

A Conformal Prediction Approach to Explore Functional Data