Abstract:Abstract Background We suggest an adaptive sample size calculation method for developing clinical prediction models, in which model performance is monitored sequentially as new data comes in. Methods We illustrate the approach using data for the diagnosis of ovarian cancer ( n = 5914, 33% event fraction) and obstructive coronary artery disease (CAD; n = 4888, 44% event fraction). We used logistic regression to develop a prediction model consisting only of a priori selected predictors and assumed linear relations for continuous predictors. We mimicked prospective patient recruitment by developing the model on 100 randomly selected patients, and we used bootstrapping to internally validate the model. We sequentially added 50 random new patients until we reached a sample size of 3000 and re-estimated model performance at each step. We examined the required sample size for satisfying the following stopping rule: obtaining a calibration slope ≥ 0.9 and optimism in the c-statistic (or AUC) < = 0.02 at two consecutive sample sizes. This procedure was repeated 500 times. We also investigated the impact of alternative modeling strategies: modeling nonlinear relations for continuous predictors and correcting for bias on the model estimates (Firth’s correction). Results Better discrimination was achieved in the ovarian cancer data (c-statistic 0.9 with 7 predictors) than in the CAD data (c-statistic 0.7 with 11 predictors). Adequate calibration and limited optimism in discrimination was achieved after a median of 450 patients (interquartile range 450–500) for the ovarian cancer data (22 events per parameter (EPP), 20–24) and 850 patients (750–900) for the CAD data (33 EPP, 30–35). A stricter criterion, requiring AUC optimism < = 0.01, was met with a median of 500 (23 EPP) and 1500 (59 EPP) patients, respectively. These sample sizes were much higher than the well-known 10 EPP rule of thumb and slightly higher than a recently published fixed sample size calculation method by Riley et al. Higher sample sizes were required when nonlinear relationships were modeled, and lower sample sizes when Firth’s correction was used. Conclusions Adaptive sample size determination can be a useful supplement to fixed a priori sample size calculations, because it allows to tailor the sample size to the specific prediction modeling context in a dynamic fashion.

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Minimum sample size for external validation of a clinical prediction model with a binary outcome

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Minimum sample size calculations for external validation of a clinical prediction model with a time‐to‐event outcome

Estimation of required sample size for external validation of risk models for binary outcomes

Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

Calculating the sample size required for developing a clinical prediction model

Clinical Prediction Models: Model Validation

Don't be misled: Three misconceptions about external validation of clinical prediction models

Evaluation of clinical prediction models (part 1): from development to external validation

Sample Size Analysis for Machine Learning Clinical Validation Studies

Adaptive sample size determination for the development of clinical prediction models

External validation of machine learning models - registered models and adaptive sample splitting

The expected value of sample information calculations for external validation of risk prediction models

A practical solution to estimate the sample size required for clinical prediction models generated from observational research on data

External validation: a simulation study to compare cross-validation versus holdout or external testing to assess the performance of clinical prediction models using PET data from DLBCL patients

Internal and external validation of predictive models: A simulation study of bias and precision in small samples

Sample size for developing a prediction model with a binary outcome: targeting precise individual risk estimates to improve clinical decisions and fairness

An evaluation of sample size requirements for developing risk prediction models with binary outcomes

External validation of a Cox prognostic model: principles and methods