Abstract:While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P <0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P =0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls

Modeling Generalization in Machine Learning: A Methodological and Computational Study

Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database

General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models

A review of model evaluation metrics for machine learning in genetics and genomics

Illusory generalizability of clinical prediction models

Prediction of Model Generalizability for Unseen Data: Methodology and Case Study in Brain Metastases Detection in T1-Weighted Contrast-Enhanced 3D MRI

Testing Generalizability in Causal Inference

Statistical Thinking, Machine Learning

Generalization in medical AI: a perspective on developing scalable models

Generalization Error of Generalized Linear Models in High Dimensions

Machine learning models and over-fitting considerations

Key Technology Considerations in Developing and Deploying Machine Learning Models in Clinical Radiology Practice

Probing out-of-distribution generalization in machine learning for materials

Evaluation Gaps in Machine Learning Practice

Shortcut learning in medical AI hinders generalization: method for estimating AI model generalization without external data

Why do probabilistic clinical models fail to transport between sites

Panoramic imaging errors in machine learning model development: a systematic review

Validity problems in clinical machine learning by indirect data labeling using consensus definitions

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Evaluating Generalizability of Deep Learning Models Using Indian-COVID-19 CT Dataset