Abstract:Abstract Background The Society of Thoracic Surgeons and European System for Cardiac Operative Risk Evaluation (EuroSCORE) II risk scores are the most commonly used risk prediction models for in-hospital mortality after adult cardiac surgery. However, they are prone to miscalibration over time and poor generalization across data sets; thus, their use remains controversial. Despite increased interest, a gap in understanding the effect of data set drift on the performance of machine learning (ML) over time remains a barrier to its wider use in clinical practice. Data set drift occurs when an ML system underperforms because of a mismatch between the data it was developed from and the data on which it is deployed. Objective In this study, we analyzed the extent of performance drift using models built on a large UK cardiac surgery database. The objectives were to (1) rank and assess the extent of performance drift in cardiac surgery risk ML models over time and (2) investigate any potential influence of data set drift and variable importance drift on performance drift. Methods We conducted a retrospective analysis of prospectively, routinely gathered data on adult patients undergoing cardiac surgery in the United Kingdom between 2012 and 2019. We temporally split the data 70:30 into a training and validation set and a holdout set. Five novel ML mortality prediction models were developed and assessed, along with EuroSCORE II, for relationships between and within variable importance drift, performance drift, and actual data set drift. Performance was assessed using a consensus metric. Results A total of 227,087 adults underwent cardiac surgery during the study period, with a mortality rate of 2.76% (n=6258). There was strong evidence of a decrease in overall performance across all models ( P <.0001). Extreme gradient boosting (clinical effectiveness metric [CEM] 0.728, 95% CI 0.728-0.729) and random forest (CEM 0.727, 95% CI 0.727-0.728) were the overall best-performing models, both temporally and nontemporally. EuroSCORE II performed the worst across all comparisons. Sharp changes in variable importance and data set drift from October to December 2017, from June to July 2018, and from December 2018 to February 2019 mirrored the effects of performance decrease across models. Conclusions All models show a decrease in at least 3 of the 5 individual metrics. CEM and variable importance drift detection demonstrate the limitation of logistic regression methods used for cardiac surgery risk prediction and the effects of data set drift. Future work will be required to determine the interplay between ML models and whether ensemble models could improve on their respective performance advantages.

Consistency of ranking was evaluated as new measure for prediction model stability: longitudinal cohort study

Stability of clinical prediction models developed using statistical or machine learning methods

Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

Updating Clinical Risk Stratification Models Using Rank-Based Compatibility: Approaches for Evaluating and Optimizing Clinician-Model Team Performance

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

A Comparative Analysis of Eight Risk Prediction Models in a High-Risk Cohort in Lithuania

Evaluation of clinical prediction models (part 1): from development to external validation

Assessing Risk Prediction Models Using Individual Participant Data from Multiple Studies

Evaluating risk prediction models for adults with heart failure: A systematic literature review

Assessing the Value of Risk Predictions by Using Risk Stratification Tables

Harnessing repeated measurements of predictor variables for clinical risk prediction: a review of existing methods

Risk prediction for prostate cancer recurrence through regularized estimation with simultaneous adjustment for nonlinear clinical effects

On the variability of regression shrinkage methods for clinical prediction models: simulation study on predictive performance

Stability Approach to Regularization Selection for Reduced-Rank Regression

Comparison of dynamic updating strategies for clinical prediction models

TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

Sample size for developing a prediction model with a binary outcome: targeting precise individual risk estimates to improve clinical decisions and fairness

Prognostic risk models for incident hypertension: A PRISMA systematic review and meta-analysis

Evaluating Discrimination of a Lung Cancer Risk Prediction Model Using Partial Risk-Score in a Two-Phase Study

Instability of Variable-selection Algorithms Used to Identify True Predictors of an Outcome in Intermediate-dimension Epidemiologic Studies

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis