Abstract:In a recent review, it was suggested that much larger cohorts are needed to prove the diagnostic value of neuroimaging biomarkers in psychiatry. While within a sample, an increase of diagnostic accuracy of schizophrenia (SZ) with number of subjects (N) has been shown, the relationship between N and accuracy is completely different between studies. Using data from a recent meta-analysis of machine learning (ML) in imaging SZ, we found that while low-N studies can reach 90% and higher accuracy, above N/2 = 50 the maximum accuracy achieved steadily drops to below 70% for N/2 > 150. We investigate the role N plays in the wide variability in accuracy results in SZ studies (63-97%). We hypothesize that the underlying cause of the decrease in accuracy with increasing N is sample heterogeneity. While smaller studies more easily include a homogeneous group of subjects (strict inclusion criteria are easily met; subjects live close to study site), larger studies inevitably need to relax the criteria/recruit from large geographic areas. A SZ prediction model based on a heterogeneous group of patients with presumably a heterogeneous pattern of structural or functional brain changes will not be able to capture the whole variety of changes, thus being limited to patterns shared by most patients. In addition to heterogeneity (sample size), we investigate other factors influencing accuracy and introduce a ML effect size. We derive a simple model of how the different factors, such as sample heterogeneity and study setup determine this ML effect size, and explain the variation in prediction accuracies found from the literature, both in cross-validation and independent sample testing. From this, we argue that smaller-N studies may reach high prediction accuracy at the cost of lower generalizability to other samples. Higher-N studies, on the other hand, will have more generalization power, but at the cost of lower accuracy. In conclusion, when comparing results from different ML studies, the sample sizes should be taken into account. To assess the generalizability of the models, validation (by direct application) of the prediction models should be tested in independent samples. The prediction of more complex measures such as outcome, which are expected to have an underlying pattern of more subtle brain abnormalities (lower effect size), will require large samples.

Quality over quantity: powering neuroimaging samples in psychiatry

Elevating the field for applying neuroimaging to individual patients in psychiatry

Detecting Neuroimaging Biomarkers for Psychiatric Disorders: Sample Size Matters

Bringing Imaging Biomarkers Into Clinical Reality in Psychiatry

The power of many brains: Catalyzing neuropsychiatric discovery through open neuroimaging data and large-scale collaboration

Small effect size leads to reproducibility failure in resting-state fMRI studies

Leveraging the adolescent brain cognitive development study to improve behavioral prediction from neuroimaging in smaller replication samples

Scanning the horizon: towards transparent and reproducible neuroimaging research

MRI economics: Balancing sample size and scan duration in brain wide association studies

Replicability and generalizability in population psychiatric neuroimaging

Systematic misestimation of machine learning performance in neuroimaging studies of depression

Psychiatric Neuroimaging at a Crossroads: Insights from Psychiatric Genetics

Big Connectome Imaging Data in Psychiatric Disorders

Towards Precision Medicine in Psychosis: Benefits and Challenges of Multimodal Multicenter Studies—PSYSCAN: Translating Neuroimaging Findings From Research into Clinical Practice

Racial and ethnic socioenvironmental inequity and neuroimaging in psychiatry: a brief review of the past and recommendations for the future

Opportunities for use of neuroimaging in de-risking drug development and improving clinical outcomes in psychiatry: an industry perspective

Detect, Quantify, and Incorporate Dataset Bias: A Neuroimaging Analysis on 12,207 Individuals

Influence of sample size and analytic approach on stability and interpretation of brain‐behavior correlations in task‐related fMRI data

Densely sampled neuroimaging for maximizing clinical insight in psychiatric and addiction disorders

How to optimize neuroscience data utilization and experiment design for advancing primate visual and linguistic brain models?

450. Symptom Severity of Sample Population Impacts the Reproducibility of Neuro-Symptom Relationship in Psychosis