Abstract:Background and objective: The minimum sample size for multistakeholder Delphi surveys remains understudied. Drawing from three large international multistakeholder Delphi surveys, this study aimed to: 1) investigate the effect of increasing sample size on replicability of results; 2) assess whether the level of replicability of results differed with participant characteristics: for example, gender, age, and profession. Methods: We used data from Delphi surveys to develop guidance for improved reporting of health-care intervention trials: SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and CONSORT (Consolidated Standards of Reporting Trials) extension for surrogate end points (n = 175, 22 items rated); CONSORT-SPI [CONSORT extension for Social and Psychological Interventions] (n = 333, 77 items rated); and core outcome set for burn care (n = 553, 88 items rated). Resampling with replacement was used to draw random subsamples from the participant data set in each of the three surveys. For each subsample, the median value of all rated survey items was calculated and compared to the medians from the full participant data set. The median number (and interquartile range) of medians replicated was used to calculate the percentage replicability (and variability). High replicability was defined as ≥80% and moderate as 60% and <80% RESULTS: The average median replicability (variability) as a percentage of total number of items rated from the three datasets was 81% (10%) at a sample size of 60. In one of the datasets (CONSORT-SPI), a ≥80% replicability was reached at a sample size of 80. On average, increasing the sample size from 80 to 160 increased the replicability of results by a further 3% and reduced variability by 1%. For subgroup analysis based on participant characteristics (eg, gender, age, professional role), using resampled samples of 20 to 100 showed that a sample size of 20 to 30 resulted to moderate replicability levels of 64% to 77%. Conclusion: We found that a minimum sample size of 60-80 participants in multistakeholder Delphi surveys provides a high level of replicability (≥80%) in the results. For Delphi studies limited to individual stakeholder groups (such as researchers, clinicians, patients), a sample size of 20 to 30 per group may be sufficient.

What are the minimal sample size requirements for Mokken scaling? An empirical example with the Warwick- Edinburgh Mental Well-Being Scale

Sufficient Sample Sizes for Multilevel Modeling

A solution to minimum sample size for regressions

No rationale for 1 variable per 10 events criterion for binary logistic regression analysis

Effect of Sample Size on Common Item Equating Using the Dichotomous Rasch Model

Exploratory Factor Analysis With Small Sample Sizes

Sample size for binary logistic prediction models: Beyond events per variable criteria

Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

Minimizing Sample Size When Using Exploratory Factor Analysis for Measurement

An evaluation of sample size requirements for developing risk prediction models with binary outcomes

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

Sample size for developing a prediction model with a binary outcome: targeting precise individual risk estimates to improve clinical decisions and fairness

Sample Size Guidelines for Logistic Regression from Observational Studies with Large Population: Emphasis on the Accuracy Between Statistics and Parameters Based on Real Life Clinical Data

Investigating psychometric properties and dimensional structure of an educational environment measure (DREEM) using Mokken scale analysis – a pragmatic approach

Collected Notes on Aldrich-Mckelevey Scaling

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Sample size requirements for stated choice experiments

Development and initial validation of the Scale of Scales (SOS) overreporting scores for the MMPI family of instruments

Sample size in multistakeholder Delphi surveys: at what minimum sample size do replicability of results stabilize?

The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration

Logistic regression models for patient-level prediction based on massive observational data: Do we need all data?