Abstract:While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P <0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P =0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

Clinical Prediction Models: Model Validation

Clinical Prediction Models: Model Development

A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models.

Clinical Prediction Models: Basic Concepts, Application Scenarios, and Research Strategies

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Evaluation of clinical prediction models (part 1): from development to external validation

Overview of model validation for survival regression model with competing risks using melanoma study data

Cardiovascular Disease Risk Prediction Models in Chinese Population- A Systematic Review and Meta-Analysis

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Don't be misled: Three misconceptions about external validation of clinical prediction models

Statistical Development and Validation of Clinical Prediction Models

Prediction Models in Prehospital and Emergency Medicine Research: How to Derive and Internally Validate a Clinical Prediction Model

External Validation of the Prognostic Prediction Model for 4-Year Risk of Metabolic Syndrome in Adults: A Retrospective Cohort Study

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

A roadmap to fair and trustworthy prediction model validation in healthcare

Minimum sample size for external validation of a clinical prediction model with a binary outcome

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Targeted Development and Validation of Clinical Prediction Models in Secondary Care Settings: Opportunities and Challenges for Electronic Health Record Data

External validation of a Cox prognostic model: principles and methods

External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

Empirical investigation of multi-source cross-validation in clinical machine learning