Abstract:Abstract Background A prediction model can be a useful tool to quantify the risk of a patient developing dementia in the next years and take risk-factor-targeted intervention. Numerous dementia prediction models have been developed, but few have been externally validated, likely limiting their clinical uptake. In our previous work, we had limited success in externally validating some of these existing models due to inadequate reporting. As a result, we are compelled to develop and externally validate novel models to predict dementia in the general population across a network of observational databases. We assess regularization methods to obtain parsimonious models that are of lower complexity and easier to implement. Methods Logistic regression models were developed across a network of five observational databases with electronic health records (EHRs) and claims data to predict 5-year dementia risk in persons aged 55–84. The regularization methods L1 and Broken Adaptive Ridge (BAR) as well as three candidate predictor sets to optimize prediction performance were assessed. The predictor sets include a baseline set using only age and sex, a full set including all available candidate predictors, and a phenotype set which includes a limited number of clinically relevant predictors. Results BAR can be used for variable selection, outperforming L1 when a parsimonious model is desired. Adding candidate predictors for disease diagnosis and drug exposure generally improves the performance of baseline models using only age and sex. While a model trained on German EHR data saw an increase in AUROC from 0.74 to 0.83 with additional predictors, a model trained on US EHR data showed only minimal improvement from 0.79 to 0.81 AUROC. Nevertheless, the latter model developed using BAR regularization on the clinically relevant predictor set was ultimately chosen as best performing model as it demonstrated more consistent external validation performance and improved calibration. Conclusions We developed and externally validated patient-level models to predict dementia. Our results show that although dementia prediction is highly driven by demographic age, adding predictors based on condition diagnoses and drug exposures further improves prediction performance. BAR regularization outperforms L1 regularization to yield the most parsimonious yet still well-performing prediction model for dementia.

External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

Clinical Prediction Models: Model Validation

There is no such thing as a validated prediction model

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

Validation of an Artificial Intelligence-Based Prediction Model Using 5 External PET/CT Datasets of Diffuse Large B-Cell Lymphoma

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Don't be misled: Three misconceptions about external validation of clinical prediction models

External validation of machine learning models - registered models and adaptive sample splitting

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Power and reproducibility in the external validation of brain-phenotype predictions

Evaluation of clinical prediction models (part 1): from development to external validation

Minimum sample size for external validation of a clinical prediction model with a binary outcome

External validation demonstrates machine learning models outperform human experts in prediction of objective and patient-reported overactive bladder treatment outcomes

Development and validation of a patient-level model to predict dementia across a network of observational databases

Performance of externally validated machine learning models based on histopathology images for the diagnosis, classification, prognosis, or treatment outcome prediction in female breast cancer: A systematic review

Experts' cognition-driven ensemble deep learning for external validation of predicting pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer

Targeted Development and Validation of Clinical Prediction Models in Secondary Care Settings: Opportunities and Challenges for Electronic Health Record Data

External validation of machine learning prediction model for pulmonary hypertension due to left heart disease

External Validation of Predictive Models for Diagnosis, Management and Severity of Pediatric Appendicitis

The Development and External Validation of Artificial Intelligence-Driven MRI-Based Models to Improve Prediction of Lesion-Specific Extraprostatic Extension in Patients with Prostate Cancer