Abstract:Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low‐dimensional data. The focus was on out‐of‐sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one‐standard‐error rule for tuning applied to cross‐validation (1SE CV) often resulted in severe miscalibration. Standard non‐repeated and repeated cross‐validation (both 5‐fold and 10‐fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross‐validation‐based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5‐fold or 10‐fold cross‐validation that minimizes out‐of‐sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non‐repeated cross‐validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low‐dimensional settings.

Machine learning models and over-fitting considerations

A novel lifelong machine learning-based method to eliminate calibration drift in clinical prediction models.

Overfitting, Model Tuning, and Evaluation of Prediction Performance

Clinical Prediction Models: Model Validation

An Optimal Solution to the Overfitting and Underfitting Problem of Healthcare Machine Learning Models

Statistical Thinking, Machine Learning

Overview of model validation for survival regression model with competing risks using melanoma study data

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications

A roadmap to fair and trustworthy prediction model validation in healthcare

Making Early Predictions of the Accuracy of Machine Learning Applications

External validation of machine learning models - registered models and adaptive sample splitting

Machine Learning for Benchmarking Critical Care Outcomes

The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study

Unraveling overoptimism and publication bias in ML-driven science

Don't Waste Your Time: Early Stopping Cross-Validation

Stabilizing Machine Learning for Reproducible and Explainable Results: A Novel Validation Approach to Subject-Specific Insights

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Practical Statistical Considerations for the Clinical Validation of AI/ML-enabled Medical Diagnostic Devices

Machine learning algorithm validation with a limited sample size