Abstract:Recent work suggests that machine learning models predicting psychiatric treatment outcomes based on clinical data may fail when applied to unharmonized samples. Neuroimaging predictive models offer the opportunity to incorporate neurobiological information, which may be more robust to dataset shifts. Yet, among the minority of neuroimaging studies that undertake any form of external validation, there is a notable lack of attention to generalization across dataset-specific idiosyncrasies. Research settings, by design, remove the between-site variations that real-world and, eventually, clinical applications demand. Here, we rigorously test the ability of a range of predictive models to generalize across three diverse, unharmonized samples: the Philadelphia Neurodevelopmental Cohort (n=1291), the Healthy Brain Network (n=1110), and the Human Connectome Project in Development (n=428). These datasets have high inter-dataset heterogeneity, encompassing substantial variations in age distribution, sex, racial and ethnic minority representation, recruitment geography, clinical symptom burdens, fMRI tasks, sequences, and behavioral measures. We demonstrate that reproducible and generalizable brain-behavior associations can be realized across diverse dataset features with sample sizes in the hundreds. Results indicate the potential of functional connectivity-based predictive models to be robust despite substantial inter-dataset variability. Notably, for the HCPD and HBN datasets, the best predictions were not from training and testing in the same dataset (i.e., cross-validation) but across datasets. This result suggests that training on diverse data may improve prediction in specific cases. Overall, this work provides a critical foundation for future work evaluating the generalizability of neuroimaging predictive models in real-world scenarios and clinical settings.

Phenotype Prediction Using BEN-Based Predictive Modeling (BPM)

Modeling Brain Functional Dynamics Via Hidden Markov Models

Bayesian Predictive Modeling Based on Multidimensional Connectivity Profiling.

Connectome-based prediction modeling of cognitive control using functional and structural connectivity

Multigroup Recognition of Dementia Patients with Dynamic Brain Connectivity under Multimodal Cortex Parcellation

Prediction of cognitive scores by movie-watching fmri connectivity and eye movement via spectral graph convolutions

The heritability and structural correlates of brain entropy

The cell-type underpinnings of regional brain entropy (BEN) in human brain

FM-APP: Foundation Model for Any Phenotype Prediction Via Fmri to Smri Knowledge Transfer

Brain-phenotype predictions can survive across diverse real-world data

Direct comparison of EEG resting state and task functional connectivity patterns for predicting working memory performance using connectome-based predictive modeling

Using modular connectome-based predictive modeling to reveal brain-behavior relationships of individual differences in working memory

The brain entropy dynamics in resting state

BrainNet with Connectivity Attention for Individualized Predictions Based on Multi-Facet Connections Extracted from Resting-State Fmri Data

Identification of brain states, transitions, and communities using functional MRI

Reliability and predictability of phenotype information from functional connectivity in large imaging datasets

Connectome-based individualized prediction of temperament trait scores

Brain-phenotype predictions of language and executive function can survive across diverse real-world data: Dataset shifts in developmental populations

432. Success and Failure of Connectome-Based Prediction of Cognitive Functioning in Early Psychosis

EEG Complexity and Functional Connectivity During Precise Timing Prediction

Prediction of Biological Motion Perception Performance from Intrinsic Brain Network Regional Efficiency