Abstract:Conformal Prediction (CP) quantifies network uncertainty by building a small prediction set with a pre-defined probability that the correct class is within this set. In this study we tackle the problem of CP calibration based on a validation set with noisy labels. We introduce a conformal score that is robust to label noise. The noise-free conformal score is estimated using the noisy labeled data and the noise level. In the test phase the noise-free score is used to form the prediction set. We applied the proposed algorithm to several standard medical imaging classification datasets. We show that our method outperforms current methods by a large margin, in terms of the average size of the prediction set, while maintaining the required coverage.
What problem does this paper attempt to address?
This paper aims to solve the problem of Conformal Prediction (CP) calibration in the presence of label noise. Specifically, the author focuses on how to apply the CP algorithm using a validation set with noisy labels in medical image classification tasks. Traditional methods, when dealing with data with noisy labels, often generate larger prediction sets and may not achieve the required coverage. This paper proposes a new conformal score, which is robust to label noise, enabling the reduction of the average size of the prediction set while maintaining the required coverage.
### Main contributions of the paper
1. **Proposing a new conformal score**: This score can estimate the noise - free conformal score, even when trained with data having noisy labels.
2. **Improving the size of the prediction set**: By introducing the noise - robust conformal score, the method in this paper can significantly reduce the size of the prediction set while maintaining high coverage.
3. **Experimental verification**: The author conducted experiments on multiple standard medical image classification datasets, and the results show that their method significantly outperforms existing methods.
### Specific methods
- **Noise model**: Assume that the label noise follows a uniform distribution, that is, the label of each sample has a probability of \(\epsilon\) to be randomly replaced with the label of other categories.
- **Noise - robust conformal score**: Estimate the noise - free conformal score \(\hat{S}(x,\tilde{y},\epsilon)\), where \(\tilde{y}\) is the observed noisy label and \(\epsilon\) is the noise level.
- **CP calibration process**: Use the estimated conformal score \(\hat{S}(x,\tilde{y},\epsilon)\) to calculate the threshold \(q_\epsilon\) and construct the prediction set accordingly.
### Experimental results
- **Datasets**: Include multiple medical image classification datasets such as TissueMNIST, PathMNIST, HAM10000 and OrganSMNIST.
- **Evaluation metrics**: Mainly evaluate the average size and coverage of the prediction set.
- **Results**: On all test datasets, the proposed NR - CP method significantly reduces the average size of the prediction set while maintaining high coverage, outperforming the Noisy - CP and NRES - CP methods.
### Conclusion
This paper proposes a new noise - robust conformal score, which can effectively apply the Conformal Prediction algorithm in the presence of label noise. The experimental results show that this method performs excellently in multiple medical image classification tasks and significantly improves the efficiency and accuracy of the prediction set.