Abstract:Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Previous works often employ temperature scaling to calibrate classifiers, assuming that confidence calibration benefits conformal prediction. However, the specific impact of confidence calibration on conformal prediction remains underexplored. In this work, we make two key discoveries about the impact of confidence calibration methods on adaptive conformal prediction. Firstly, we empirically show that current confidence calibration methods (e.g., temperature scaling) typically lead to larger prediction sets in adaptive conformal prediction. Secondly, by investigating the role of temperature value, we observe that high-confidence predictions can enhance the efficiency of adaptive conformal prediction. Theoretically, we prove that predictions with higher confidence result in smaller prediction sets on expectation. This finding implies that the rescaling parameters in these calibration methods, when optimized with cross-entropy loss, might counteract the goal of generating efficient prediction sets. To address this issue, we propose Conformal Temperature Scaling (ConfTS), a variant of temperature scaling with a novel loss function designed to enhance the efficiency of prediction sets. This approach can be extended to optimize the parameters of other post-hoc methods of confidence calibration. Extensive experiments demonstrate that our method improves existing adaptive conformal prediction methods in classification tasks, especially with LLMs.

Entropy Reweighted Conformal Classification

Conformal Prediction via Regression-as-Classification

Trustworthy Classification through Rank-Based Conformal Prediction Sets

An Information Theoretic Perspective on Conformal Prediction

Efficient Conformal Prediction under Data Heterogeneity

Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration

Improving Uncertainty Quantification of Deep Classifiers via Neighborhood Conformal Prediction: Novel Algorithm and Theoretical Analysis

Conformal Prediction: A Data Perspective

Weighted Aggregation of Conformity Scores for Classification

Does confidence calibration improve conformal prediction?

Group-Weighted Conformal Prediction

Conformal Prediction for Deep Classifier via Label Ranking

conformalClassification: A Conformal Prediction R Package for Classification

Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

Provably Robust Conformal Prediction with Improved Efficiency

Robust Yet Efficient Conformal Prediction Sets

Generalization and Informativeness of Conformal Prediction

Adapting Conformal Prediction to Distribution Shifts Without Labels

A Conformal Prediction Score that is Robust to Label Noise

C-Adapter: Adapting Deep Classifiers for Efficient Conformal Prediction Sets

Adaptive Conformal Prediction by Reweighting Nonconformity Score