Abstract:Studies of cardiovascular disease risk prediction by machine learning algorithms often do not assess their ability to generalize to other populations and few of them include an analysis of the interpretability of individual predictions. This manuscript addresses the development and validation, both internal and external, of predictive models for the assessment of risks of major adverse cardiovascular events (MACE). Global and local interpretability analyses of predictions were conducted towards improving MACE's model reliability and tailoring preventive interventions. The models were trained and validated on a retrospective cohort with the use of data from Ribeirão Preto Medical School (RPMS), University of São Paulo, Brazil. Data from Beth Israel Deaconess Medical Center (BIDMC), USA, were used for external validation. A balanced sample of 6,000 MACE cases and 6,000 non-MACE cases from RPMS was created for training and internal validation and an additional one of 8,000 MACE cases and 8,000 non-MACE cases from BIDMC was employed for external validation. Eight machine learning algorithms, namely Penalized Logistic Regression, Random Forest, XGBoost, Decision Tree, Support Vector Machine, k-Nearest Neighbors, Naive Bayes, and Multi-Layer Perceptron were trained to predict a 5-year risk of major adverse cardiovascular events and their predictive performance was evaluated regarding accuracy, ROC curve (receiver operating characteristic), and AUC (area under the ROC curve). LIME and Shapley values were applied towards insights about model interpretability. Random Forest showed the best predictive performance in both internal validation (AUC = 0.871 (0.859–0.882); Accuracy = 0.794 (0.782–0.808)) and external one (AUC = 0.786 (0.778–0.792); Accuracy = 0.710 (0.704–0.717)). Compared to LIME, Shapley values suggest more consistent explanations on exploratory analysis and importance of features. Among the machine learning algorithms evaluated, Random Forest showed the best generalization ability, both internally and externally. Shapley values for local interpretability were more informative than LIME ones, which is in line with our exploratory analysis and global interpretation of the final model. Machine learning algorithms with good generalization and accompanied by interpretability analyses are recommended for assessments of individual risks of cardiovascular diseases and development of personalized preventive actions.

Machine learning models' assessment: trust and performance

Comparison of Machine Learning Models and Framingham Risk Score for the prediction of the presence and severity of Coronary Artery Diseases by using Gensini Score

A New Approach for Interpretability and Reliability in Clinical Risk Prediction: Acute Coronary Syndrome Scenario

Machine learning-based risk prediction for major adverse cardiovascular events in a Brazilian hospital: Development, external validation, and interpretability

Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review

Comparative analysis of explainable machine learning prediction models for hospital mortality

Improving Cardiovascular Risk Prediction Through Machine Learning Modelling of Irregularly Repeated Electronic Health Records

Evaluating Machine Learning Models against Clinical Protocols for Enhanced Interpretability and Continuity of Care

Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning

Enhancing Cardiovascular Disease Risk Prediction with Machine Learning Models

An interpretable machine learning method for risk stratification of patients with acute coronary syndrome

Health Risk Assessment Using Machine Learning: Systematic Review

An interpretable data-driven approach for rules construction: Application to cardiovascular risk assessment

A Machine Learning Approach Employing Common Clinical Variables

Disseminating the Risk Factors With Enhancement in Precision Medicine Using Comparative Machine Learning Models for Healthcare Data

Fusion of risk assessment models with application to coronary artery disease patients

A Human-Machine Trust Model Integrating Machine Estimated Performance

Monitoring machine learning (ML)-based risk prediction algorithms in the presence of confounding medical interventions

Machine-Learning based Prediction Models for Healthcare Outcomes in Patients Participating in Cardiac Rehabilitation: A Systematic Review

A systematic literature review on Machine Learning Model evaluation on healthcare applications

Using Machine Learning to Improve the Prediction of Functional Outcome in Ischemic Stroke Patients