Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning
Chenxi HuangShu-Xia LiCésar CaraballoFrederick A. MasoudiJohn S. RumsfeldJohn A. SpertusSharon-Lise T. NormandBobak J. MortazaviHarlan M. KrumholzCenter for Outcomes Research and Evaluation,Yale New Haven Hospital,CT (C.H.,S.-X.L.,C.C.,H.M.K.).Division of Cardiology,Unversity of Colorado Anschutz Medical Campus,Aurora,CO (F.A.M.,J.S.R.).Ascension Health,St Louis,MO (F.A.M.)Department of Internal Medicine,University of Missouri,Kansas City,MO (J.A.S.).Department of Cardiovascular Medicine,Saint Luke's Mid America Heart Institute,Kansas City,MO (J.A.S.)Department of Biostatistics,T.H. Chan School of Public Health,Harvard University,Boston,MA (S.-L.T.N.).Department of Health Care Policy,Harvard Medical School,Boston,MA (S.-L.T.N.).Department of Computer Science & Engineering,Texas A&M University,College Station (B.J.M.).Department of Health Policy and Management,Yale School of Public Health New Haven,CT (H.M.K.).Section of Cardiovascular Medicine,Department of Internal Medicine,Yale School of Medicine,New Haven,CT (H.M.K.).
DOI: https://doi.org/10.1161/CIRCOUTCOMES.120.007526
2021-10-04
Abstract:New methods such as machine learning techniques have been increasingly used to enhance the performance of risk predictions for clinical decision-making. However, commonly reported performance metrics may not be sufficient to capture the advantages of these newly proposed models for their adoption by health care professionals to improve care. Machine learning models often improve risk estimation for certain subpopulations that may be missed by these metrics. This article addresses the limitations of commonly reported metrics for performance comparison and proposes additional metrics. Our discussions cover metrics related to overall performance, discrimination, calibration, resolution, reclassification, and model implementation. Models for predicting acute kidney injury after percutaneous coronary intervention are used to illustrate the use of these metrics. We demonstrate that commonly reported metrics may not have sufficient sensitivity to identify improvement of machine learning models and propose the use of a comprehensive list of performance metrics for reporting and comparing clinical risk prediction models.