Abstract:Risk prediction models fitted using maximum likelihood estimation (MLE) are often overfitted resulting in predictions that are too extreme and a calibration slope (CS) less than 1. Penalized methods, such as Ridge and Lasso, have been suggested as a solution to this problem as they tend to shrink regression coefficients toward zero, resulting in predictions closer to the average. The amount of shrinkage is regulated by a tuning parameter, λ, commonly selected via cross‐validation ("standard tuning"). Though penalized methods have been found to improve calibration on average, they often over‐shrink and exhibit large variability in the selected λ and hence the CS. This is a problem, particularly for small sample sizes, but also when using sample sizes recommended to control overfitting. We consider whether these problems are partly due to selecting λ using cross‐validation with "training" datasets of reduced size compared to the original development sample, resulting in an over‐estimation of λ and, hence, excessive shrinkage. We propose a modified cross‐validation tuning method ("modified tuning"), which estimates λ from a pseudo‐development dataset obtained via bootstrapping from the original dataset, albeit of larger size, such that the resulting cross‐validation training datasets are of the same size as the original dataset. Modified tuning can be easily implemented in standard software and is closely related to bootstrap selection of the tuning parameter ("bootstrap tuning"). We evaluated modified and bootstrap tuning for Ridge and Lasso in simulated and real data using recommended sample sizes, and sizes slightly lower and higher. They substantially improved the selection of λ , resulting in improved CS compared to the standard tuning method. They also improved predictions compared to MLE.

Improvements on cross-validation: the 632+ bootstrap method

Bootstrapping the Cross-Validation Estimate

Cross-validation: what does it estimate and how well does it do it?

Bootstrapping the Out-of-sample Predictions for Efficient and Accurate Cross-Validation

On The Smoothness of Cross-Validation-Based Estimators Of Classifier Performance

Bootstrap Cross-validation Improves Model Selection in Pharmacometrics

Is Cross-Validation the Gold Standard to Evaluate Model Performance?

Estimation of prediction error by using K-fold cross-validation

Penalized Regression Methods With Modified Cross‐Validation and Bootstrap Tuning Produce Better Prediction Models

Approximate Cross-validation: Guarantees for Model Assessment and Selection

The difference of model robustness assessment using cross‐validation and bootstrap methods

A survey of cross-validation procedures for model selection

Selecting a classification method by cross-validation

Iterative Approximate Cross-Validation

Subsampling Bias and The Best-Discrepancy Systematic Cross Validation

Don't Waste Your Time: Early Stopping Cross-Validation

Performance-Estimation Properties of Cross-Validation-Based Protocols with Simultaneous Hyper-Parameter Optimization

Confidence intervals centered on bootstrap smoothed estimators

On confidence intervals centered on bootstrap smoothed estimators

Confidence intervals for the Cox model test error from cross-validation

Is K-fold cross validation the best model selection method for Machine Learning?