Abstract:Multi-modal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domains-specific, leading to data being represented in dissimilar subspaces, and (ii) fused data is often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multi-modal artificial intelligence (AI) model with a novel fusion module inspired by a bottleneck attention module (BAM). We use deep neural networks (DNN) to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multi-modal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (P < 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multi-modal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to schizophrenia. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.

Multi-SIGATnet: A multimodal schizophrenia MRI classification algorithm using sparse interaction mechanisms and graph attention networks

Exploring Schizophrenia Classification Through Multimodal MRI and Deep Graph Neural Networks: Unveiling Brain Region-Specific Weight Discrepancies and Their Association With Cell-Type Specific Transcriptomic Features

Schizophrenia Identification Using Multi-View Graph Measures of Functional Brain Networks

Multi-modal deep learning from imaging genomic data for schizophrenia classification

Discriminative analysis of schizophrenia patients using graph convolutional networks: A combined multimodal MRI and connectomics analysis

MMM: Classification of Schizophrenia Using Multi-Modality Multi-Atlas Feature Representation and Multi-Kernel Learning.

Assisting Schizophrenia Diagnosis Using Clinical Electroencephalography and Interpretable Graph Neural Networks: a Real-World and Cross-Site Study

Multimodal Analysis of Structural and Functional MRI for Schizophrenia Diagnosis

Classification of Schizophrenia Based on Graph Product Depth Neural Network Fusion of Fmri and Dmri Multidimensional Information

Classification of schizophrenia patients using a graph convolutional network: A combined functional MRI and connectomics analysis

Recognition of the Multi-class Schizophrenia Based on the Resting-State EEG Network Topology

Discriminative Analysis of Schizophrenia Patients Using Topological Properties of Structural and Functional Brain Networks: A Multimodal Magnetic Resonance Imaging Study

Multi-Modal Diagnosis of Alzheimer's Disease using Interpretable Graph Convolutional Networks

Classification of EEG-Based Brain Connectivity Networks in Schizophrenia Using a Multi-Domain Connectome Convolutional Neural Network

Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders

Multi feature fusion network for schizophrenia classification and abnormal brain network recognition

A brain structure learning-guided multi-view graph representation learning for brain network analysis

Multimodal classification of drug-naïve first-episode schizophrenia combining anatomical, diffusion and resting state functional resonance imaging.

Sparsity‐guided multiple functional connectivity patterns for classification of schizophrenia via convolutional network

Multi-Scale Spatial-Temporal Attention Networks for Functional Connectome Classification

Multi-modal Imaging Genomics Transformer: Attentive Integration of Imaging with Genomic Biomarkers for Schizophrenia Classification