Abstract:Abstract In recent years, the collection and sharing of resting-state functional magnetic resonance imaging (fMRI) datasets across multiple centers have enabled studying psychiatric disorders at scale, and prompted the application of statistically powerful tools such as deep neural networks. Yet, multi-center datasets introduce non-biological heterogeneity that can confound the biological signal of interest and produce erroneous findings. To mitigate this problem, the neuroimaging community has adopted harmonization techniques previously proposed in other domains to remove site-effects from fMRI data. The reported success of these approaches in improving the generalization of the models have varied significantly. It remains unclear whether harmonization techniques could boost the final outcome of multi-site fMRI studies, to what extent, and which approaches are best suited for this task. In an attempt to objectively answer these questions, we conduct a standardized rigorous evaluation of seven different harmonization techniques from the neuroimaging and computer vision literature on two large-scale multi-site datasets ( N = 2169 and N = 2366) to diagnose autism spectrum disorder and major depression disorder from static and dynamic representations of fMRI data. Interestingly, while all harmonization techniques removed site-effects from the data, they had little influence on disorder classification performance in standard k-fold and leave-one-site-out validation settings over a well-tuned baseline. Further investigation shows that the baseline model implicitly learns site-invariant features which could well explain its competitiveness with explicit harmonization techniques and suggest orthogonality between latent disease features and site discrminative features. However, additional experiments show that harmonization methods could be critical to report faithful results in settings where there is high intra-site class imbalance and the learning algorithm is prone to overfit on spurious features confounding the final outcome of the study.

Harmonizing the CBCL and SDQ ADHD scores by using linear equating, kernel equating, item response theory and machine learning methods

Harmonisation of assessments of attention, social, emotional, and behaviour problems using the Child Behavior Checklist and the Strengths and Difficulties Questionnaire

Harmonizing two measures of adaptive functioning using computational approaches: prediction of vineland adaptive behavior scales II (VABS-II) from the adaptive behavior assessment system II (ABAS-II) scores

Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

The 2014 Ontario Child Health Study Emotional Behavioural Scales (OCHS-EBS) Part II: Psychometric Adequacy for Categorical Measurement of Selected DSM-5 Disorders

Equalização das escalas NESSCA e SARA utilizando a Teoria da Resposta ao Item na avaliação do comprometimento pela doença de Machado-Joseph

Psychometric evaluation of a brief measure to capture general population-level variation in ADHD symptoms from childhood through the transition to adulthood

Enhancing comparability in early child development assessment with the D-score

New Robust Scale Transformation Methods in the Presence of Outlying Common Items.

Cross-distribution Comparison of Five linear Equating Methods Based on Mixed Model of Item Response Theory

Measurement invariance of the Child Behavior Checklist (CBCL) across race/ethnicity and sex in the Adolescent Brain and Cognitive Development (ABCD) study.

Evaluating Robust Scale Transformation Methods with Multiple Outlying Common Items under IRT True Score Equating.

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

Systematic Integration of Multi-Informant Externalizing Ratings in Clinical Settings

Concordance and validity between versions of the ADHD Conners scale for Parents

Harmonization techniques for machine learning studies using multi-site functional MRI data

Using Bayesian item response theory for multicohort repeated measure design to estimate individual latent change scores.

Avoiding bias from sum scores in growth estimates: An examination of IRT-based approaches to scoring longitudinal survey responses.

Using a Linear Regression Method to Detect Outliers in IRT Common Item Equating

The Strengths and Difficulties Questionnaire as a Predictor of Parent-Reported Diagnosis of Autism Spectrum Disorder and Attention Deficit Hyperactivity Disorder

Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models.