Abstract:Importance: Risk estimation is an integral part of cardiovascular care. Local recalibration of guideline-recommended models could address the limitations of existing tools. Objective: To provide a machine learning (ML) approach to augment the performance of the American Heart Association's Predicting Risk of Cardiovascular Disease Events (AHA-PREVENT) equations when applied to a local population while preserving clinical interpretability. Design, setting, and participants: This cohort study used a New England-based electronic health record cohort of patients without prior atherosclerotic cardiovascular disease (ASCVD) who had the data necessary to calculate the AHA-PREVENT 10-year risk of developing ASCVD in the event period (2007-2016). Patients with prior ASCVD events, death prior to 2007, or age 79 years or older in 2007 were subsequently excluded. The final study population of 95 326 patients was split into 3 nonoverlapping subsets for training, testing, and validation. The AHA-PREVENT model was adapted to this local population using the open-source ML model (MLM) Extreme Gradient Boosting model (XGBoost) with minimal predictor variables, including age, sex, and AHA-PREVENT. The MLM was monotonically constrained to preserve known associations between risk factors and ASCVD risk. Along with sex, race and ethnicity data from the electronic health record were collected to validate the performance of ASCVD risk prediction in subgroups. Data were analyzed from August 2021 to February 2024. Main outcomes and measures: Consistent with the AHA-PREVENT model, ASCVD events were defined as the first occurrence of either nonfatal myocardial infarction, coronary artery disease, ischemic stroke, or cardiovascular death. Cardiovascular death was coded via government registries. Discrimination, calibration, and risk reclassification were assessed using the Harrell C index, a modified Hosmer-Lemeshow goodness-of-fit test and calibration curves, and reclassification tables, respectively. Results: In the test set of 38 137 patients (mean [SD] age, 64.8 [6.9] years, 22 708 [59.5]% women and 15 429 [40.5%] men; 935 [2.5%] Asian, 2153 [5.6%] Black, 1414 [3.7%] Hispanic, 31 400 [82.3%] White, and 2235 [5.9%] other, including American Indian, multiple races, unspecified, and unrecorded, consolidated owing to small numbers), MLM-PREVENT had improved calibration (modified Hosmer-Lemeshow P > .05) compared to the AHA-PREVENT model across risk categories in the overall cohort (χ23 = 2.2; P = .53 vs χ23 > 16.3; P < .001) and sex subgroups (men: χ23 = 2.1; P = .55 vs χ23 > 16.3; P < .001; women: χ23 = 6.5; P = .09 vs. χ23 > 16.3; P < .001), while also surpassing a traditional recalibration approach. MLM-PREVENT maintained or improved AHA-PREVENT's calibration in Asian, Black, and White individuals. Both MLM-PREVENT and AHA-PREVENT performed equally well in discriminating risk (approximate ΔC index, ±0.01). Using a clinically significant 7.5% risk threshold, MLM-PREVENT reclassified a total of 11.5% of patients. We visualize the recalibration through MLM-PREVENT ASCVD risk charts that highlight preserved risk associations of the original AHA-PREVENT model. Conclusions and relevance: The interpretable ML approach presented in this article enhanced the accuracy of the AHA-PREVENT model when applied to a local population while still preserving the risk associations found by the original model. This method has the potential to recalibrate other established risk tools and is implementable in electronic health record systems for improved cardiovascular risk assessment.

Calibration plots for multistate risk predictions models

Calibration plots for multistate risk predictions models: an overview and simulation comparing novel approaches

A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models.

Risk prediction models for discrete ordinal outcomes: calibration and the impact of the proportional odds assumption

Non‐parametric inference on calibration of predicted risks

Assessing the calibration of subdistribution hazard models in discrete time

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

Overview of model validation for survival regression model with competing risks using melanoma study data

Fair admission risk prediction with proportional multicalibration

Emulator-Based Bayesian Calibration of the CISNET Colorectal Cancer Models

A constrained maximum likelihood approach to developing well-calibrated models for predicting binary outcomes

Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

A novel calibration framework for survival analysis when a binary covariate is measured at sparse time points

Decision Curve Analysis: a Technical Note

Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees

Joint calibrated estimation of inverse probability of treatment and censoring weights for marginal structural models

An evaluation of sample size requirements for developing risk prediction models with binary outcomes

Tailoring Risk Prediction Models to Local Populations

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

The harms of class imbalance corrections for machine learning based prediction models: a simulation study

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches