Conformal Loss-Controlling Prediction

Di Wang,Ping Wang,Zhong Ji,Xiaojun Yang,Hongyue Li

2024-01-23

Abstract:Conformal prediction is a learning framework controlling prediction coverage of prediction sets, which can be built on any learning algorithm for point prediction. This work proposes a learning framework named conformal loss-controlling prediction, which extends conformal prediction to the situation where the value of a loss function needs to be controlled. Different from existing works about risk-controlling prediction sets and conformal risk control with the purpose of controlling the expected values of loss functions, the proposed approach in this paper focuses on the loss for any test object, which is an extension of conformal prediction from miscoverage loss to some general loss. The controlling guarantee is proved under the assumption of exchangeability of data in finite-sample cases and the framework is tested empirically for classification with a class-varying loss and statistical postprocessing of numerical weather forecasting applications, which are introduced as point-wise classification and point-wise regression problems. All theoretical analysis and experimental results confirm the effectiveness of our loss-controlling approach.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to control the loss value of any test object during the prediction process, rather than just controlling the coverage rate of the prediction set or the expected loss value. Traditional Conformal Prediction (CP) methods mainly focus on controlling the coverage rate of the prediction set, that is, controlling the miscoverage loss, but these methods cannot be directly applied to applications that need to control a wider range of loss types. For example, in medical image classification or multi - label classification tasks, the consequences caused by different categories may be different, so the loss value needs to be adjusted according to the specific category. In addition, in applications such as tumor segmentation, in addition to over - covering pixels, more attention needs to be paid to controlling other types of losses, such as the false - negative rate.

To this end, the paper proposes a new framework named Conformal Loss - Controlling Prediction (CLCP), which extends traditional conformal prediction and enables it to control the specific loss value of any test object, rather than just its expected value. CLCP provides a theoretical control guarantee by using the assumption of data exchangeability in the case of a finite sample, ensuring that the predicted loss does not exceed the preset loss level \(\alpha\) with high probability. This method is not only applicable to classification problems, but can also be applied to regression problems such as numerical weather forecasting, demonstrating its wide applicability and effectiveness.