Abstract:OBJECTIVE:Clinical prediction models (CPMs) constructed based on artificial intelligence have been proven to have positive impacts on clinical activities. However, the deterioration of CPM performance over time has rarely been studied. This paper proposes a model updating method to solve the calibration drift issue caused by data drift. MATERIALS AND METHODS:This paper proposes a novel model updating method based on lifelong machine learning (LML). The effectiveness of the proposed method is verified in four tumor datasets, and a comprehensive comparison with other model updating methods is performed. RESULTS:Changes in data distributions cause model performances to drift. The four compared model updating methods have different effects in terms of improving the discrimination and calibration abilities of the tested models. The LML method proposed in this study improves model performance better than or equivalent to the other methods. The proposed method achieved a mean AUC of 0.8249, 0.8780, 0.8261, and 0.8489, a mean AUPRC of 0.7782, 0.9730, 0.4655, and 0.5728, a mean F1 of 0.6866, 0.9552, 0.2985, and 0.3585, and a mean estimated calibration index (ECI) of 0.0320, 0.0338, 0.0101, and 0.0115 using colorectal, lung, breast and prostate cancer datasets. DISCUSSION:The LML framework simultaneously monitors model performance and the distribution of disease risk characteristics, enabling it to effectively address the performance degradation caused by gradual and sudden data drifts and provide reasonable explanations for the causes of performance degradation. CONCLUSION:Monitoring model performance and the underlying data distribution can promote model life cycle iteration with "development-deployment-maintenance-monitoring" as the core, which, in turn, ensures that the model can provide accurate predictions, guides the model update process and explains the causes of model performance changes.

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

Learn then test: Calibrating predictive algorithms to achieve risk control

A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models.

T-Cal: An optimal test for the calibration of predictive models

Automatically Adaptive Conformal Risk Control

Cautious Calibration in Binary Classification

Probabilistic Safety Regions Via Finite Families of Scalable Classifiers

Loss-Controlling Calibration for Predictive Models

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Images

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Non‐parametric inference on calibration of predicted risks

Fair admission risk prediction with proportional multicalibration

Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks

A Comparative Study of Confidence Calibration in Deep Learning: From Computer Vision to Medical Imaging

Risk-based Calibration for Probabilistic Classifiers

Calibrated Selective Classification

Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions

Minimum-Risk Recalibration of Classifiers

Distribution-free, risk-controlling prediction sets

Testing Calibration in Nearly-Linear Time