Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

Yue Cai,Yu-Qing Cai,Li-Ying Tang,Yi-Han Wang,Mengchun Gong,Tian-Ci Jing,Hui-Jun Li,Jesse Li-Ling,Wei Hu,Zhihua Yin,Da-Xin Gong,Guang-Wei Zhang

DOI: https://doi.org/10.1186/s12916-024-03273-7

IF: 9.3

2024-02-07

BMC Medicine

Abstract:A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

medicine, general & internal

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the lack of comprehensive reviews and independent external validation tools for artificial intelligence (AI) models in cardiovascular disease (CVD) risk prediction. Specifically, the goals of the paper include:

1. **Identification, Description, and Evaluation**: Systematically identify, describe, and evaluate AI models (AI-Ms) used for cardiovascular disease risk prediction, particularly in the general population and specific populations.
2. **Development of Independent Validation Scoring System**: Develop a new independent validation scoring (IVS) system to assess the reproducibility of AI models.

### Background and Significance

Cardiovascular disease has become a global challenge, with mortality increasing from 12.1 million in 1990 to 18.6 million in 2019. Risk prediction is one of the main strategies to address this issue and has achieved significant results in some developed countries. Traditional risk prediction models (such as the Framingham and SCORE models) have been incorporated into clinical guidelines, but these models have certain limitations.

Artificial intelligence (AI), particularly machine learning (ML) and deep learning (DL), has rapidly developed in the medical field, showing excellent performance in disease risk prediction. However, despite the great potential of AI in CVD prediction, there is currently a lack of comprehensive and systematic reviews in this field, especially regarding independent external validation.

### Main Findings

1. **Study Design and Population**:
- Included 79 articles, most (82.25%) published between 2017 and 2021.
- Used 114 datasets, mainly from Europe and North America, with no datasets from Africa.
- Identified 486 AI models, most of which are in the development stage and have not undergone independent external validation.

2. **Predictors**:
- The median number of predictors was 21, ranging from 5 to 52,000.
- In addition to traditional factors (such as age, gender, total cholesterol, and smoking status), new predictors emerged, such as electrocardiogram images, ultrasound images, and magnetic resonance imaging.

3. **Model Performance**:
- Most models had sample sizes between 1,000 and 10,000, with a median of 4,466.
- Reported various performance metrics, such as C-index, calibration plots, sensitivity, and specificity.

4. **Algorithm Transparency and Model Reproducibility**:
- Identified 66 different algorithms, with the most commonly used being logistic regression, random forest, and neural networks.
- Most models (212) did not report code, formulas, or hyperparameters, thus considered non-reproducible.

5. **Independent External Validation**:
- All models had a high risk of bias, mainly due to inappropriate statistical methods.
- Developed a new IVS system, with results showing only 10 models recommended, 281 models not recommended, and 187 models flagged with warnings.

### Conclusion

AI is leading a digital revolution in the field of CVD risk prediction but is still in the early stages of development, with deficiencies in study design, reporting, and evaluation systems. The developed IVS system aids in independent external validation and the advancement of this field. Future research should focus more on multi-center and multi-country data sources to improve the external validation and clinical application value of the models.