Abstract:Multimodal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of subjects' underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domain specific, leading to data being represented in dissimilar subspaces, and (ii) fused data are often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multimodal artificial intelligence model with a novel fusion module inspired by a bottleneck attention module. We use deep neural networks to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multimodal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (p < .0001), outperforming state-of-the-art unimodal and multimodal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network's decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multimodal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to SZ. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.

Multimodal integration of neuroimaging and genetic data for the diagnosis of mood disorders based on computer vision models

Multimodal integration of neuroimaging and genetic data for the diagnosis of mood disorders based on computer vision models

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

A Joint Study of Whole Exome Sequencing and Structural MRI Analysis in Major Depressive Disorder.

Identification and discovery of imaging genetic patterns using fusion self-expressive network in major depressive disorder

Identification of multimodal brain imaging biomarkers in first-episode drugs-naive major depressive disorder through a multi-site large-scale MRI consortium data

Deep Multimodal Predictome for Studying Mental Disorders

An attention-based multi-modal MRI fusion model for major depressive disorder diagnosis

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness

Incorporating multi-stage diagnosis status to mine associations between genetic risk variants and the multi-modality phenotype network in major depressive disorder.

Adaptive Multimodal Neuroimage Integration for Major Depression Disorder Detection

Multimodal fusion of structural and functional brain imaging in depression using linked independent component analysis

Machine Learning Models Identify Multimodal Measurements Highly Predictive of Transdiagnostic Symptom Severity for Mood, Anhedonia, and Anxiety

Imaging-genomic Spatial-Modality Attentive Fusion for Studying Neuropsychiatric Disorders.

Automatic Diagnosis of Major Depressive Disorder Using a High- and Low-Frequency Feature Fusion Framework

Predictive Brain Networks for Major Depression in a Semi-Multimodal Fusion Hierarchical Feature Reduction Framework

Enhancing Early Diagnosis of Bipolar Disorder in Adolescents through Multimodal Neuroimaging

Distinguish Bipolar and Major Depressive Disorder in Adolescents Based on Multimodal Neuroimaging: Results from the Adolescent Brain Cognitive Development Study®.

Diagnosis of Major Depressive Disorder Using Machine Learning Based on Multisequence MRI Neuroimaging Features