Abstract:Neuropsychiatric disorders such as schizophrenia are very heterogeneous in nature and typically diagnosed using self-reported symptoms. This makes it difficult to pose a confident prediction on the cases and does not provide insight into the underlying neural and biological mechanisms of these disorders. Combining neuroimaging and genomic data with a multi-modal 'predictome' paves the way for biologically informed markers and may improve prediction reliability. With that, we develop a multi-modal deep learning framework by fusing data from different modalities to capture the interaction between the latent features and evaluate their complementary information in characterizing schizophrenia. Our deep model uses structural MRI, functional MRI, and genome-wide polymorphism data to perform the classification task. It includes a multi-layer feed-forward network, an encoder, and a long short-term memory (LSTM) unit with attention to learn the latent features and adopt a joint training scheme capturing synergies between the modalities. The hybrid network also uses different regularizers for addressing the inherent overfitting and modality-specific bias in the multi-modal setup. Next, we run the network through a saliency model to analyze the learned features. Integrating modalities enhances the performance of the classifier, and our framework acquired 88% (P < 0.0001) accuracy on a dataset of 437 subjects. The trimodal accuracy is comparable to the state-of-the-art performance on a data collection of this size and outperforms the unimodal and bimodal baselines we compared. Model introspection was used to expose the salient neural features and genes/biological pathways associated with schizophrenia. To our best knowledge, this is the first approach that fuses genomic information with structural and functional MRI biomarkers for predicting schizophrenia. We believe this type of modality blending can better explain the disorder's dynamics by adding cross-modal prospects.Clinical Relevance- This study combinedly learns imaging and genomic features for the classification of schizophrenia. The data fusion scheme extracts modality interactions, and the saliency experiments report multiple functional and structural networks closely connected to the disorder.

A generative-discriminative framework that integrates imaging, genetic, and diagnosis into coupled low dimensional space

A Partially Functional Linear Modeling Framework for Integrating Genetic, Imaging, and Clinical Data

Generative discriminative models for multivariate inference and statistical mapping in medical imaging

Joint Sparse Collaborative Regression on Imaging Genetics Study of Schizophrenia.

An interpretable generative multimodal neuroimaging-genomics framework for decoding Alzheimer's disease

Identifying disease sensitive and quantitative trait-relevant biomarkers from multidimensional heterogeneous imaging genetics data via sparse multimodal multitask learning.

A Deep-Generative Hybrid Model to Integrate Multimodal and Dynamic Connectivity for Predicting Spectrum-Level Deficits in Autism

Achieving multi-modal brain disease diagnosis performance using only single-modal images through generative AI

Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness

Multi-modal deep learning from imaging genomic data for schizophrenia classification

Transcriptome-guided Amyloid Imaging Genetic Analysis Via a Novel Structured Sparse Learning Algorithm.

Deep sr-DDL: Deep structurally regularized dynamic dictionary learning to integrate multimodal and dynamic functional connectomics data for multidimensional clinical characterizations

Multimodal and multicontrast image fusion via deep generative models

A deep generative multimodal imaging genomics framework for Alzheimer's disease prediction

Integration of Imaging (epi)genomics Data for the Study of Schizophrenia Using Group Sparse Joint Nonnegative Matrix Factorization

Gene-SGAN: discovering disease subtypes with imaging and genetic signatures via multi-view weakly-supervised deep clustering

Gene-SGAN: a method for discovering disease subtypes with imaging and genetic signatures via multi-view weakly-supervised deep clustering

Integrating Multimodal Neuroimaging and Genetics: A Structurally-Linked Sparse Canonical Correlation Analysis Approach

Diagnosis Status Guided Brain Imaging Genetics Via Integrated Regression and Sparse Canonical Correlation Analysis.

Integration of Multiple Genomic Imaging Data for the Study of Schizophrenia Using Joint Nonnegative Matrix Factorization.

Deep Multimodal Predictome for Studying Mental Disorders