Abstract:In a recent review, it was suggested that much larger cohorts are needed to prove the diagnostic value of neuroimaging biomarkers in psychiatry. While within a sample, an increase of diagnostic accuracy of schizophrenia (SZ) with number of subjects (N) has been shown, the relationship between N and accuracy is completely different between studies. Using data from a recent meta-analysis of machine learning (ML) in imaging SZ, we found that while low-N studies can reach 90% and higher accuracy, above N/2 = 50 the maximum accuracy achieved steadily drops to below 70% for N/2 > 150. We investigate the role N plays in the wide variability in accuracy results in SZ studies (63-97%). We hypothesize that the underlying cause of the decrease in accuracy with increasing N is sample heterogeneity. While smaller studies more easily include a homogeneous group of subjects (strict inclusion criteria are easily met; subjects live close to study site), larger studies inevitably need to relax the criteria/recruit from large geographic areas. A SZ prediction model based on a heterogeneous group of patients with presumably a heterogeneous pattern of structural or functional brain changes will not be able to capture the whole variety of changes, thus being limited to patterns shared by most patients. In addition to heterogeneity (sample size), we investigate other factors influencing accuracy and introduce a ML effect size. We derive a simple model of how the different factors, such as sample heterogeneity and study setup determine this ML effect size, and explain the variation in prediction accuracies found from the literature, both in cross-validation and independent sample testing. From this, we argue that smaller-N studies may reach high prediction accuracy at the cost of lower generalizability to other samples. Higher-N studies, on the other hand, will have more generalization power, but at the cost of lower accuracy. In conclusion, when comparing results from different ML studies, the sample sizes should be taken into account. To assess the generalizability of the models, validation (by direct application) of the prediction models should be tested in independent samples. The prediction of more complex measures such as outcome, which are expected to have an underlying pattern of more subtle brain abnormalities (lower effect size), will require large samples.

450. Symptom Severity of Sample Population Impacts the Reproducibility of Neuro-Symptom Relationship in Psychosis

Small effect size leads to reproducibility failure in resting-state fMRI studies

Detecting Neuroimaging Biomarkers for Psychiatric Disorders: Sample Size Matters

Decoding Early Psychoses: Unraveling Stable Microstructural Features Associated with Psychopathology Across Independent Cohorts

Replicability and generalizability in population psychiatric neuroimaging

Quality over quantity: powering neuroimaging samples in psychiatry

Effect Sizes of Associations Between Neuroimaging Measures and Affective Symptoms: A Meta‐analysis

P580. Two Schizophrenia Neuroanatomical Signatures from the PHENOM Consortium and Their Association with Psychopathology, Cognition, and Genetics in the Population-Level Samples

Reproducibility in the Absence of Selective Reporting: an Illustration from Large-Scale Brain Asymmetry Research

Schizophrenia Imaging Signatures and Their Associations with Cognition, Psychopathology, and Genetics in the General Population.

Replicable brain-phenotype associations require large-scale neuroimaging data

Scanning the horizon: towards transparent and reproducible neuroimaging research

An illustration of reproducibility in neuroscience research in the absence of selective reporting

Robust Brain Correlates of Cognitive Performance in Psychosis and its Prodrome

Bias in data-driven estimates of the reproducibility of univariate brain-wide association studies.

Unravelling Robust Brain-Behavior Links of Depressive Symptoms Through Granular Network Models: Understanding Heterogeneity and Clinical Implications

Replication of a neuroimaging biomarker for striatal dysfunction in psychosis

Evidence of Discontinuity Between Psychosis-Risk and Non-Clinical Samples in the Neuroanatomical Correlates of Social Function.

Neuroimaging Defined Psychosis Spectrum Phenotypes in the General Population

Systematic misestimation of machine learning performance in neuroimaging studies of depression