External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting
Anum Zahra,Maarten van Smeden,Evertine J. Abbink,Jesse M. van den Berg,Marieke T. Blom,Carline J. van den Dries,Jacobijn Gussekloo,Fenne Wouters,Karlijn J. Joling,René Melis,Simon P. Mooijaart,Jeannette B. Peters,Harmke A. Polinder-Bos,Bas F.M. van Raaij,Brent Appelman,Hannah M. la Roi-Teeuw,Karel G.M. Moons,Kim Luijken
DOI: https://doi.org/10.1016/j.jclinepi.2024.111270
IF: 7.407
2024-02-04
Journal of Clinical Epidemiology
Abstract:Objective To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three healthcare settings: hospitals, primary care, and nursing homes. Design Retrospective external validation study Setting Hospital, primary care, and nursing home settings. Participants 14,092 older individuals of ≥70 years of age with a clinical or PCR-confirmed COVID-19 diagnosis from March 2020 to December 2020. Data sources Six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (JGPN/ANH/AHA, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Model selection and eligibility criterion Based on a living systematic review of COVID-19 prediction models using PROBAST for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, NEWS2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts. The GAL-COVID-19 mortality model was validated in all three healthcare settings. Methods Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation. Main outcome measure In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting. Results All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24 to 0.81, and c-statistic 0.55 to 0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across healthcare settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24 to 0.81 indicating signs of overfitting, and c-statistic of 0.55 to 0.71. Conclusions Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each healthcare setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.
public, environmental & occupational health,health care sciences & services