Abstract:Multi-modal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domains-specific, leading to data being represented in dissimilar subspaces, and (ii) fused data is often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multi-modal artificial intelligence (AI) model with a novel fusion module inspired by a bottleneck attention module (BAM). We use deep neural networks (DNN) to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multi-modal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (P < 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multi-modal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to schizophrenia. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.

SpaDE: Semantic Locality Preserving Biclustering for Neuroimaging Data

BDEC:Brain Deep Embedded Clustering model

SpaRG: Sparsely Reconstructed Graphs for Generalizable fMRI Analysis

Deep Semantic Architecture with discriminative feature visualization for neuroimage analysis

Sparse Deep Neural Network for Encoding and Decoding the Structural Connectome

Striatal Subdivisions Estimated via Deep Embedded Clustering With Application to Parkinson's Disease

A Deep Probabilistic Spatiotemporal Framework for Dynamic Graph Representation Learning with Application to Brain Disorder Identification

N-BiC: A Method for Multi-Component and Symptom Biclustering of Structural MRI Data: Application to Schizophrenia.

Decoding autism: Uncovering patterns in brain connectivity through sparsity analysis with rs-fMRI data

Disentangle the group and individual components of functional connectome with autoencoders

Constrain Latent Space for Schizophrenia Classification via Dual Space Mapping Net

Deep sr-DDL: Deep structurally regularized dynamic dictionary learning to integrate multimodal and dynamic functional connectomics data for multidimensional clinical characterizations

Multimodal active subspace analysis for computing assessment oriented subspaces from neuroimaging data

Identifying Alzheimer's Disease-Related Brain Regions from Multi-Modality Neuroimaging Data Using Sparse Composite Linear Discrimination Analysis.

Bayesian estimation of clustered dependence structures in functional neuroconnectivity

Temporal Dynamic Synchronous Functional Brain Network for Schizophrenia Diagnosis and Lateralization Analysis

Temporal Dynamic Synchronous Functional Brain Network for Schizophrenia Classification and Lateralization Analysis

Multi-faceted Neuroimaging Data Integration via Analysis of Subspaces

Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders

Extracting Brain Disease-Related Connectome Subgraphs by Adaptive Dense Subgraph Discovery

Hybrid parcellation mapping approach for the extraction of connectivity measures in autism spectrum disorder fMRI data