Abstract:Background: The performance of a classification algorithm eventually reaches a point of diminishing returns, where the additional sample added does not improve the results. Thus, there is a need to determine an optimal sample size that maximizes performance while accounting for computational burden or budgetary concerns. Objective: This study aimed to determine optimal sample sizes and the relationships between sample size and dataset-level characteristics over a variety of binary classification algorithms. Methods: A total of 16 large open-source datasets were collected, each containing a binary clinical outcome. Furthermore, 4 machine learning algorithms were assessed: XGBoost (XGB), random forest (RF), logistic regression (LR), and neural networks (NNs). For each dataset, the cross-validated area under the curve (AUC) was calculated at increasing sample sizes, and learning curves were fit. Sample sizes needed to reach the observed full–dataset AUC minus 2 points (0.02) were calculated from the fitted learning curves and compared across the datasets and algorithms. Dataset–level characteristics, minority class proportion, full–dataset AUC, number of features, type of features, and degree of nonlinearity were examined. Negative binomial regression models were used to quantify relationships between these characteristics and expected sample sizes within each algorithm. A total of 4 multivariable models were constructed, which selected the best-fitting combination of dataset–level characteristics. Results: Among the 16 datasets (full-dataset sample sizes ranging from 70,000-1,000,000), median sample sizes were 9960 (XGB), 3404 (RF), 696 (LR), and 12,298 (NN) to reach AUC stability. For all 4 algorithms, more balanced classes (multiplier: 0.93-0.96 for a 1% increase in minority class proportion) were associated with decreased sample size. Other characteristics varied in importance across algorithms—in general, more features, weaker features, and more complex relationships between the predictors and the response increased expected sample sizes. In multivariable analysis, the top selected predictors were minority class proportion among all 4 algorithms assessed, full–dataset AUC (XGB, RF, and NN), and dataset nonlinearity (XGB, RF, and NN). For LR, the top predictors were minority class proportion, percentage of strong linear features, and number of features. Final multivariable sample size models had high goodness-of-fit, with dataset–level predictors explaining a majority (66.5%-84.5%) of the total deviance in the data among all 4 models. Conclusions: The sample sizes needed to reach AUC stability among 4 popular classification algorithms vary by dataset and method and are associated with dataset–level characteristics that can be influenced or estimated before the start of a research study.

Evaluation of a decided sample size in machine learning applications

Sample-Size Determination Methodologies for Machine Learning in Medical Imaging Research: A Systematic Review

Sample Size Analysis for Machine Learning Clinical Validation Studies

Toward Generalizable Machine Learning Models in Speech, Language, and Hearing Sciences: Estimating Sample Size and Reducing Overfitting

Determining Sample Size in Binary Measurement System

Sample Size Requirements for Popular Classification Algorithms in Tabular Clinical Data: Empirical Study

A refined approach for evaluating small datasets via binary classification using machine learning

Sample size estimation for power and accuracy in the experimental comparison of algorithms

Small sample size effects in statistical pattern recognition: recommendations for practitioners

An evaluation of sample size requirements for developing risk prediction models with binary outcomes

Machine learning algorithm validation with a limited sample size

Sample size determination for multidimensional parameters and the A-optimal subsampling in a big data linear regression model

How to calculate sample size in animal and human studies

All about sample-size calculations for A/B testing: Novel extensions and practical guide

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Sample Size in Natural Language Processing within Healthcare Research

Systematic Bias in Sample Inference and its Effect on Machine Learning

Sample size for developing a prediction model with a binary outcome: targeting precise individual risk estimates to improve clinical decisions and fairness

Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances