Abstract:Background Data drift can negatively impact the performance of machine learning algorithms (MLAs) that were trained on historical data. As such, MLAs should be continuously monitored and tuned to overcome the systematic changes that occur in the distribution of data. In this paper, we study the extent of data drift and provide insights about its characteristics for sepsis onset prediction. This study will help elucidate the nature of data drift for prediction of sepsis and similar diseases. This may aid with the development of more effective patient monitoring systems that can stratify risk for dynamic disease states in hospitals. Methods We devise a series of simulations that measure the effects of data drift in patients with sepsis. We simulate multiple scenarios in which data drift may occur, namely the change in the distribution of the predictor variables (covariate shift), the change in the statistical relationship between the predictors and the target (concept shift), and the occurrence of a major healthcare event (major event) such as the COVID-19 pandemic. We measure the impact of data drift on model performances, identify the circumstances that necessitate model retraining, and compare the effects of different retraining methodologies and model architecture on the outcomes. We present the results for two different MLAs, eXtreme Gradient Boosting (XGB) and Recurrent Neural Network (RNN). Results Our results show that the properly retrained XGB models outperform the baseline models in all simulation scenarios, hence signifying the existence of data drift. In the major event scenario, the area under the receiver operating characteristic curve (AUROC) at the end of the simulation period is 0.811 for the baseline XGB model and 0.868 for the retrained XGB model. In the covariate shift scenario, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.853 and 0.874 respectively. In the concept shift scenario and under the mixed labeling method, the retrained XGB models perform worse than the baseline model for most simulation steps. However, under the full relabeling method, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.852 and 0.877 respectively. The results for the RNN models were mixed, suggesting that retraining based on a fixed network architecture may be inadequate for an RNN. We also present the results in the form of other performance metrics such as the ratio of observed to expected probabilities (calibration) and the normalized rate of positive predictive values (PPV) by prevalence, referred to as lift, at a sensitivity of 0.8. Conclusion Our simulations reveal that retraining periods of a couple of months or using several thousand patients are likely to be adequate to monitor machine learning models that predict sepsis. This indicates that a machine learning system for sepsis prediction will probably need less infrastructure for performance monitoring and retraining compared to other applications in which data drift is more frequent and continuous. Our results also show that in the event of a concept shift, a full overhaul of the sepsis prediction model may be necessary because it indicates a discrete change in the definition of sepsis labels, and mixing the labels for the sake of incremental training may not produce the desired results.

Modeling sepsis progression using hidden Markov models

Machine-Learning-Enabled Prognostic Models for Sepsis

Identifying and analyzing sepsis states: A retrospective study on patients with sepsis in ICUs

Hidden Markov Models for sepsis detection in preterm infants

A machine learning based predictive model for the diagnosis of sepsis

Subtle variation in sepsis-III definitions markedly influences predictive performance within and across methods

Prognostic Models for Sepsis Built on Small Datasets

Enhancing Patient Selection in Sepsis Clinical Trials Design Through an AI Enrichment Strategy: Algorithm Development and Validation

Early prediction of sepsis using a high-order Markov dynamic Bayesian network (HMDBN) classifier

A Quantitative Model For Sepsis Stratification

An interpretable machine learning model for real-time sepsis prediction based on basic physiological indicators

Frequent temporal patterns of physiological and biological biomarkers and their evolution in sepsis

MGP-AttTCN: An Interpretable Machine Learning Model for the Prediction of Sepsis

Identification and Validation of Sepsis Subphenotypes Using Time-Series Data

Artificial neural networks for disease trajectory prediction in the context of sepsis

Platelet Metabolites as Candidate Biomarkers in Sepsis Diagnosis and Management Using the Proposed Explainable Artificial Intelligence Approach

Real-Time Prediction of Sepsis in Critical Trauma Patients: Machine Learning–Based Modeling Study

Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction

An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection

An application based on bioinformatics and machine learning for risk prediction of sepsis at first clinical presentation using transcriptomic data

Advanced Meta-Ensemble Machine Learning Models for Early and Accurate Sepsis Prediction to Improve Patient Outcomes