Abstract:OBJECTIVE:Clinical prediction models (CPMs) constructed based on artificial intelligence have been proven to have positive impacts on clinical activities. However, the deterioration of CPM performance over time has rarely been studied. This paper proposes a model updating method to solve the calibration drift issue caused by data drift. MATERIALS AND METHODS:This paper proposes a novel model updating method based on lifelong machine learning (LML). The effectiveness of the proposed method is verified in four tumor datasets, and a comprehensive comparison with other model updating methods is performed. RESULTS:Changes in data distributions cause model performances to drift. The four compared model updating methods have different effects in terms of improving the discrimination and calibration abilities of the tested models. The LML method proposed in this study improves model performance better than or equivalent to the other methods. The proposed method achieved a mean AUC of 0.8249, 0.8780, 0.8261, and 0.8489, a mean AUPRC of 0.7782, 0.9730, 0.4655, and 0.5728, a mean F1 of 0.6866, 0.9552, 0.2985, and 0.3585, and a mean estimated calibration index (ECI) of 0.0320, 0.0338, 0.0101, and 0.0115 using colorectal, lung, breast and prostate cancer datasets. DISCUSSION:The LML framework simultaneously monitors model performance and the distribution of disease risk characteristics, enabling it to effectively address the performance degradation caused by gradual and sudden data drifts and provide reasonable explanations for the causes of performance degradation. CONCLUSION:Monitoring model performance and the underlying data distribution can promote model life cycle iteration with "development-deployment-maintenance-monitoring" as the core, which, in turn, ensures that the model can provide accurate predictions, guides the model update process and explains the causes of model performance changes.

How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection

A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models.

Enhancing Model Adaptability Using Concept Drift Detection for Short-Term Load Forecast

Time to Retrain? Detecting Concept Drifts in Machine Learning Systems

Efficiently Mitigating the Impact of Data Drift on Machine Learning Pipelines

Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study

Towards Computational Performance Engineering for Unsupervised Concept Drift Detection -- Complexities, Benchmarking, Performance Analysis

Expert-Driven Monitoring of Operational ML Models

Automating concept-drift detection by self-evaluating predictive model degradation

A Scalable Approach to Covariate and Concept Drift Management via Adaptive Data Segmentation

On The Reliability Of Machine Learning Applications In Manufacturing Environments

LEAF: Navigating Concept Drift in Cellular Networks

On the Reliable Detection of Concept Drift from Streaming Unlabeled Data

Monitoring Machine Learning Models: Online Detection of Relevant Deviations

A Model-Driven Engineering Approach for Monitoring Machine Learning Models

A survey on detecting healthcare concept drift in AI/ML models from a finance perspective

Identifying Predictions That Influence the Future: Detecting Performative Concept Drift in Data Streams

Unsupervised Concept Drift Detection from Deep Learning Representations in Real-time

Machine Learning Model Drift Detection Via Weak Data Slices

Handling Concept Drifts in Regression Problems -- the Error Intersection Approach

Detecting and Responding to Concept Drift in Business Processes