Jenna M Reps,Chungsoo Kim,Ross D Williams,Aniek F Markus,Cynthia Yang,Talita Duarte-Salles,Thomas Falconer,Jitendra Jonnagaddala,Andrew Williams,Sergio Fernández-Bertolín,Scott L DuVall,Kristin Kostka,Gowtham Rao,Azza Shoaibi,Anna Ostropolets,Matthew E Spotnitz,Lin Zhang,Paula Casajust,Ewout W Steyerberg,Fredrik Nyberg,Benjamin Skov Kaas-Hansen,Young Hwa Choi,Daniel Morales,Siaw-Teng Liaw,Maria Tereza Fernandes Abrahão,Carlos Areia,Michael E Matheny,Kristine E Lynch,María Aragón,Rae Woong Park,George Hripcsak,Christian G Reich,Marc A Suchard,Seng Chan You,Patrick B Ryan,Daniel Prieto-Alhambra,Peter R Rijnbeek

Abstract:Background SARS-CoV-2 is straining health care systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate patients who require hospitalization from those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision-making during the pandemic. However, the model is at high risk of bias according to the “prediction model risk of bias assessment” criteria, and it has not been externally validated. Objective The aim of this study was to externally validate the C-19 index across a range of health care settings to determine how well it broadly predicts hospitalization due to pneumonia in COVID-19 cases. Methods We followed the Observational Health Data Sciences and Informatics (OHDSI) framework for external validation to assess the reliability of the C-19 index. We evaluated the model on two different target populations, 41,381 patients who presented with SARS-CoV-2 at an outpatient or emergency department visit and 9,429,285 patients who presented with influenza or related symptoms during an outpatient or emergency department visit, to predict their risk of hospitalization with pneumonia during the following 0-30 days. In total, we validated the model across a network of 14 databases spanning the United States, Europe, Australia, and Asia. Results The internal validation performance of the C-19 index had a C statistic of 0.73, and the calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data, the model obtained C statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US, and South Korean data sets, respectively. The calibration was poor, with the model underestimating risk. When validated on 12 data sets containing influenza patients across the OHDSI network, the C statistics ranged between 0.40 and 0.68. Conclusions Our results show that the discriminative performance of the C-19 index model is low for influenza cohorts and even worse among patients with COVID-19 in the United States, Spain, and South Korea. These results suggest that C-19 should not be used to aid decision-making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.

External validation of two clinical prediction models for pediatric pneumonia

Development and Internal Validation of a Prediction Model to Risk Stratify Children With Suspected Community-Acquired Pneumonia

Predicting Severe Pneumonia Outcomes in Children

Derivation and validation of a clinical prediction model for risk-stratification of children hospitalized with severe pneumonia in Bangladesh

External Validation of Predictive Models for Diagnosis, Management and Severity of Pediatric Appendicitis

Predicting Hospitalised Paediatric Pneumonia Mortality Risk: An External Validation of RISC and mRISC, and Local Tool Development (RISC-Malawi) from Malawi

Development of a Nomogram for Predicting Refractory Mycoplasma pneumoniae Pneumonia in Children

Evaluation of clinical prediction models (part 1): from development to external validation

Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study

Construction and Verification of a Predictive Model for Risk Factors in Children With Severe Adenoviral Pneumonia

External validation of the computer aided risk scoring system in predicting in-hospital mortality following emergency medical admissions

Prediction Models for Bronchopulmonary Dysplasia in Preterm Infants: A Systematic Review

Interpretable modeling and discovery of key predictors for pneumonia diagnosis in children based on electronic medical records

Development of a new prognostic model to predict pneumonia outcome using artificial intelligence-based chest radiograph results

Implementation of the COVID-19 Vulnerability Index Across an International Network of Health Care Data Sets: Collaborative External Validation Study

A three-step diagnosis of pediatric pneumonia at the emergency department using clinical predictors, C-reactive protein, and pneumococcal PCR

Massive external validation of a machine learning algorithm to predict pulmonary embolism in hospitalized patients

Predicting nosocomial pneumonia risk in level-1 trauma patients: An external validation study using the trauma quality improvement program

A Model to Predict In-hospital Mortality in Elderly Patients with Community-acquired Pneumonia: a Retrospective Study

Comparing accuracy of clinical prediction rules to predict pneumonia in children and adolescents with acute febrile respiratory illness

Performance of Multiple Risk Assessment Tools to Predict Mortality for Adult Respiratory Distress Syndrome with Extracorporeal Membrane Oxygenation Therapy: An External Validation Study Based on Chinese Single-center Data.