Abstract:OBJECTIVE:Clinical prediction models (CPMs) constructed based on artificial intelligence have been proven to have positive impacts on clinical activities. However, the deterioration of CPM performance over time has rarely been studied. This paper proposes a model updating method to solve the calibration drift issue caused by data drift. MATERIALS AND METHODS:This paper proposes a novel model updating method based on lifelong machine learning (LML). The effectiveness of the proposed method is verified in four tumor datasets, and a comprehensive comparison with other model updating methods is performed. RESULTS:Changes in data distributions cause model performances to drift. The four compared model updating methods have different effects in terms of improving the discrimination and calibration abilities of the tested models. The LML method proposed in this study improves model performance better than or equivalent to the other methods. The proposed method achieved a mean AUC of 0.8249, 0.8780, 0.8261, and 0.8489, a mean AUPRC of 0.7782, 0.9730, 0.4655, and 0.5728, a mean F1 of 0.6866, 0.9552, 0.2985, and 0.3585, and a mean estimated calibration index (ECI) of 0.0320, 0.0338, 0.0101, and 0.0115 using colorectal, lung, breast and prostate cancer datasets. DISCUSSION:The LML framework simultaneously monitors model performance and the distribution of disease risk characteristics, enabling it to effectively address the performance degradation caused by gradual and sudden data drifts and provide reasonable explanations for the causes of performance degradation. CONCLUSION:Monitoring model performance and the underlying data distribution can promote model life cycle iteration with "development-deployment-maintenance-monitoring" as the core, which, in turn, ensures that the model can provide accurate predictions, guides the model update process and explains the causes of model performance changes.

Post deployment recycling of machine learning models

A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models.

Reusing Deep Neural Network Models through Model Re-engineering

Machine learning in recycling business: an investigation of its practicality, benefits and future trends

Model Reuse with Domain Knowledge

An Efficient Model Maintenance Approach for MLOps

Cost-Effective Retraining of Machine Learning Models

Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization

Catastrophic Forgetting in the Context of Model Updates

Sequential algorithmic modification with test data reuse

Machine Learning Approaches in Agile Manufacturing with Recycled Materials for Sustainability

Deep Learning for Fixed Model Reuse.

RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively

Predictive precision in battery recycling: unveiling lithium battery recycling potential through machine learning

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

Post-Hoc Reversal: Are We Selecting Models Prematurely?

Effective ML Model Versioning in Edge Networks

If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs

Cost-aware retraining for machine learning

On Using Retrained and Incremental Machine Learning for Modeling Performance of Adaptable Software: An Empirical Comparison

Reusable MLOps: Reusable Deployment, Reusable Infrastructure and Hot-Swappable Machine Learning models and services