Abstract:Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low‐dimensional data. The focus was on out‐of‐sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one‐standard‐error rule for tuning applied to cross‐validation (1SE CV) often resulted in severe miscalibration. Standard non‐repeated and repeated cross‐validation (both 5‐fold and 10‐fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross‐validation‐based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5‐fold or 10‐fold cross‐validation that minimizes out‐of‐sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non‐repeated cross‐validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low‐dimensional settings.

Tuning model parameters in class‐imbalanced learning with precision‐recall curve

Precision-Recall Curve (PRC) Classification Trees

Binormal Precision–Recall Curves for Optimal Classification of Imbalanced Data

The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

The receiver operating characteristic curve accurately assesses imbalanced datasets

Precision and Recall Reject Curves for Classification

Commonly used software tools produce conflicting and overly-optimistic AUPRC values

Empirical analysis of performance assessment for imbalanced classification

Classification performance assessment for imbalanced multiclass data

A measure oriented training scheme for imbalanced classification problems

Population and Empirical PR Curves for Assessment of Ranking Algorithms

Revisiting Precision and Recall Definition for Generative Model Evaluation

The Effect of Class Imbalance on Precision-Recall Curves

The MCC-F1 curve: a performance evaluation technique for binary classification

A Closer Look at AUROC and AUPRC under Class Imbalance

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study

Multiclass ROC

A Study of Data Pre-processing Techniques for Imbalanced Biomedical Data Classification

Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functions

Enabling Precision/Recall Preferences for Semi-supervised SVM Training.