Abstract:Multi-modal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domains-specific, leading to data being represented in dissimilar subspaces, and (ii) fused data is often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multi-modal artificial intelligence (AI) model with a novel fusion module inspired by a bottleneck attention module (BAM). We use deep neural networks (DNN) to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multi-modal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (P < 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multi-modal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to schizophrenia. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.

Cross-Modal Attention for Multimodal Information Fusion: A Novel Approach to Attention Deficit Hyperactivity Disorder Detection

Fusion of Multi-Task Neurophysiological Data to Enhance the Detection of Attention- Deficit/Hyperactivity Disorder

Insights into Detecting Adult ADHD Symptoms Through Advanced Dual-Stream Machine Learning

Machine learning-enabled detection of attention-deficit/hyperactivity disorder with multimodal physiological data: a case-control study

ADHD Intelligent Auxiliary Diagnosis System Based on Multimodal Information Fusion.

An Ensemble Deep Learning Approach Combining Phenotypic Data and fMRI for ADHD Diagnosis

A Virtual-Reality System Integrated With Neuro-Behavior Sensing for Attention-Deficit/Hyperactivity Disorder Intelligent Assessment

Integrating Multimodal MRIs for Adult ADHD Identification with Heterogeneous Graph Attention Convolutional Network

Action-Based ADHD Diagnosis in Video

Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

Deep Spatio-Temporal Representation and Ensemble Classification for Attention Deficit/Hyperactivity Disorder

iPAL: A Machine Learning Based Smart Healthcare Framework For Automatic Diagnosis Of Attention Deficit/Hyperactivity Disorder (ADHD)

The Hybrid Deep Learning Model for Identification of Attention-Deficit/Hyperactivity Disorder Using EEG

A Novel Audio-Visual Information Fusion System for Mental Disorders Detection

Multimodal Kernel-based discriminant correlation analysis data-fusion approach: an automated autism spectrum disorder diagnostic system

A feature-aware multimodal framework with auto-fusion for Alzheimer's disease diagnosis

Enhanced ADHD classification through deep learning and dynamic resting state fMRI analysis

Exploiting the Brain's Network Structure for Automatic Identification of ADHD Subjects

Multimodal assessment of adult attention-deficit hyperactivity disorder: A controlled virtual seminar room study

A multi-view convolutional neural network method combining attention mechanism for diagnosing autism spectrum disorder