Adaptive Conformal Prediction by Reweighting Nonconformity Score

Salim I. Amoukou,Nicolas J.B Brunel

2023-06-01

Abstract:Despite attractive theoretical guarantees and practical successes, Predictive Interval (PI) given by Conformal Prediction (CP) may not reflect the uncertainty of a given model. This limitation arises from CP methods using a constant correction for all test points, disregarding their individual uncertainties, to ensure coverage properties. To address this issue, we propose using a Quantile Regression Forest (QRF) to learn the distribution of nonconformity scores and utilizing the QRF's weights to assign more importance to samples with residuals similar to the test point. This approach results in PI lengths that are more aligned with the model's uncertainty. In addition, the weights learnt by the QRF provide a partition of the features space, allowing for more efficient computations and improved adaptiveness of the PI through groupwise conformalization. Our approach enjoys an assumption-free finite sample marginal and training-conditional coverage, and under suitable assumptions, it also ensures conditional coverage. Our methods work for any nonconformity score and are available as a Python package. We conduct experiments on simulated and real-world data that demonstrate significant improvements compared to existing methods.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that the existing Predictive Interval (PI) methods cannot well reflect the uncertainty of a given model. Specifically, the traditional Conformal Prediction (CP) method uses a fixed correction term when constructing the predictive interval, which ignores the individual differences between different test points, resulting in the width of the predictive interval not accurately reflecting the model's uncertainty at these points. This limitation stems from the fact that the CP method adopts the same correction strategy for all test points in order to ensure the coverage property, without considering the specific situation of each test point.

To solve this problem, the author proposes an adaptive conformal prediction method based on Reweighted Nonconformity Scores. By using the Quantile Regression Forest (QRF) to learn the distribution of non - conformity scores and using the weights of QRF to assign higher importance to samples with similar residuals to the test points, the length of the predictive interval is made more in line with the model's uncertainty. In addition, the weights learned by QRF also provide a partition of the feature space, which helps to calculate more effectively and improves the adaptability of the predictive interval through within - group conformal adjustment. This method not only has coverage under the no - assumption finite - sample margin and training conditions, but also can ensure conditional coverage under appropriate assumptions. The author verifies through experiments that this method has significant improvements over existing methods on both simulated data and real data.