Abstract:Background: The performance of a classification algorithm eventually reaches a point of diminishing returns, where the additional sample added does not improve the results. Thus, there is a need to determine an optimal sample size that maximizes performance while accounting for computational burden or budgetary concerns. Objective: This study aimed to determine optimal sample sizes and the relationships between sample size and dataset-level characteristics over a variety of binary classification algorithms. Methods: A total of 16 large open-source datasets were collected, each containing a binary clinical outcome. Furthermore, 4 machine learning algorithms were assessed: XGBoost (XGB), random forest (RF), logistic regression (LR), and neural networks (NNs). For each dataset, the cross-validated area under the curve (AUC) was calculated at increasing sample sizes, and learning curves were fit. Sample sizes needed to reach the observed full–dataset AUC minus 2 points (0.02) were calculated from the fitted learning curves and compared across the datasets and algorithms. Dataset–level characteristics, minority class proportion, full–dataset AUC, number of features, type of features, and degree of nonlinearity were examined. Negative binomial regression models were used to quantify relationships between these characteristics and expected sample sizes within each algorithm. A total of 4 multivariable models were constructed, which selected the best-fitting combination of dataset–level characteristics. Results: Among the 16 datasets (full-dataset sample sizes ranging from 70,000-1,000,000), median sample sizes were 9960 (XGB), 3404 (RF), 696 (LR), and 12,298 (NN) to reach AUC stability. For all 4 algorithms, more balanced classes (multiplier: 0.93-0.96 for a 1% increase in minority class proportion) were associated with decreased sample size. Other characteristics varied in importance across algorithms—in general, more features, weaker features, and more complex relationships between the predictors and the response increased expected sample sizes. In multivariable analysis, the top selected predictors were minority class proportion among all 4 algorithms assessed, full–dataset AUC (XGB, RF, and NN), and dataset nonlinearity (XGB, RF, and NN). For LR, the top predictors were minority class proportion, percentage of strong linear features, and number of features. Final multivariable sample size models had high goodness-of-fit, with dataset–level predictors explaining a majority (66.5%-84.5%) of the total deviance in the data among all 4 models. Conclusions: The sample sizes needed to reach AUC stability among 4 popular classification algorithms vary by dataset and method and are associated with dataset–level characteristics that can be influenced or estimated before the start of a research study.

Logistic regression models for patient-level prediction based on massive observational data: Do we need all data?

Adaptive sample size determination for the development of clinical prediction models

Calculating the sample size required for developing a clinical prediction model

Sample size for developing a prediction model with a binary outcome: targeting precise individual risk estimates to improve clinical decisions and fairness

Sample size for binary logistic prediction models: Beyond events per variable criteria

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

A practical solution to estimate the sample size required for clinical prediction models generated from observational research on data

Sample Size Guidelines for Logistic Regression from Observational Studies with Large Population: Emphasis on the Accuracy Between Statistics and Parameters Based on Real Life Clinical Data

Sample size considerations and predictive performance of multinomial logistic prediction models

Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Design matters in patient-level prediction: evaluation of a cohort vs. case-control design when developing predictive models in observational healthcare datasets

Estimation of required sample size for external validation of risk models for binary outcomes

An evaluation of sample size requirements for developing risk prediction models with binary outcomes

Predicting outcomes at the individual patient level: what is the best method?

Distributed Learning from Multi-Site Observational Health Data for Zero-Inflated Count Outcomes

Minimum sample size for external validation of a clinical prediction model with a binary outcome

A comparison of approaches to improve worst-case predictive model performance over patient subpopulations

Detecting Neuroimaging Biomarkers for Psychiatric Disorders: Sample Size Matters