Abstract:In a recent review, it was suggested that much larger cohorts are needed to prove the diagnostic value of neuroimaging biomarkers in psychiatry. While within a sample, an increase of diagnostic accuracy of schizophrenia (SZ) with number of subjects (N) has been shown, the relationship between N and accuracy is completely different between studies. Using data from a recent meta-analysis of machine learning (ML) in imaging SZ, we found that while low-N studies can reach 90% and higher accuracy, above N/2 = 50 the maximum accuracy achieved steadily drops to below 70% for N/2 > 150. We investigate the role N plays in the wide variability in accuracy results in SZ studies (63-97%). We hypothesize that the underlying cause of the decrease in accuracy with increasing N is sample heterogeneity. While smaller studies more easily include a homogeneous group of subjects (strict inclusion criteria are easily met; subjects live close to study site), larger studies inevitably need to relax the criteria/recruit from large geographic areas. A SZ prediction model based on a heterogeneous group of patients with presumably a heterogeneous pattern of structural or functional brain changes will not be able to capture the whole variety of changes, thus being limited to patterns shared by most patients. In addition to heterogeneity (sample size), we investigate other factors influencing accuracy and introduce a ML effect size. We derive a simple model of how the different factors, such as sample heterogeneity and study setup determine this ML effect size, and explain the variation in prediction accuracies found from the literature, both in cross-validation and independent sample testing. From this, we argue that smaller-N studies may reach high prediction accuracy at the cost of lower generalizability to other samples. Higher-N studies, on the other hand, will have more generalization power, but at the cost of lower accuracy. In conclusion, when comparing results from different ML studies, the sample sizes should be taken into account. To assess the generalizability of the models, validation (by direct application) of the prediction models should be tested in independent samples. The prediction of more complex measures such as outcome, which are expected to have an underlying pattern of more subtle brain abnormalities (lower effect size), will require large samples.

Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

Beyond out-of-sample: robust and generalizable multivariate neuroanatomical patterns of psychiatric problems in youth

Cross-species machine learning improves diagnostic classification of human psychiatric disorders

Bias in machine learning models can be significantly mitigated by careful training: Evidence from neuroimaging studies

Assisting Schizophrenia Diagnosis Using Clinical Electroencephalography and Interpretable Graph Neural Networks: a Real-World and Cross-Site Study

Supervised machine learning for diagnostic classification from large-scale neuroimaging datasets

Effects of Randomness in the Development of Machine Learning Models in Neuroimaging Studies of Schizophrenia

Brain-phenotype predictions can survive across diverse real-world data

External Validation of a Machine Learning Model for Schizophrenia Classification

Systematic misestimation of machine learning performance in neuroimaging studies of depression

Detecting Neuroimaging Biomarkers for Psychiatric Disorders: Sample Size Matters

Effects of Brain Atlases and Machine Learning Methods on the Discrimination of Schizophrenia Patients: A Multimodal MRI Study

Limited generalizability of multivariate brain-based dimensions of child psychiatric symptoms

Diagnosing schizophrenia using deep learning: Novel interpretation approaches and multi-site validation

Neuroimaging Defined Psychosis Spectrum Phenotypes in the General Population

Integrating machining learning and multimodal neuroimaging to detect schizophrenia at the level of the individual

Unveiling the potential of machine learning in schizophrenia diagnosis: A meta-analytic study of task-based neuroimaging data

Machine learning in small sample neuroimaging studies: Novel measures for schizophrenia analysis

Multi-modal deep learning from imaging genomic data for schizophrenia classification

More reliable biomarkers and more accurate prediction for mental disorders using a label-noise filtering-based dimensional prediction method

Neuroimage Analysis Methods and Artificial Intelligence Techniques for Reliable Biomarkers and Accurate Diagnosis of Schizophrenia: Achievements Made by Chinese Scholars Around the Past Decade