Abstract:Multi-modal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domains-specific, leading to data being represented in dissimilar subspaces, and (ii) fused data is often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multi-modal artificial intelligence (AI) model with a novel fusion module inspired by a bottleneck attention module (BAM). We use deep neural networks (DNN) to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multi-modal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (P < 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multi-modal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to schizophrenia. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.

Exploring Latent Constructs through Multimodal Data Analysis

The Development of a Multidimensional Diagnostic Assessment With Learning Tools to Improve 3-D Mental Rotation Skills

Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models

Cognitive Diagnosis Modeling Incorporating Response Times and Fixation Counts: Providing Comprehensive Feedback and Accurate Diagnosis

Cognitive Diagnosis Modelling Incorporating Response Times and Fixation Counts: Providing A More Comprehensive and Accurate Diagnosis of Problem-Solving

Latent Space Explorer: Visual Analytics for Multimodal Latent Space Exploration

Latent Feature Extraction for Process Data Via Multidimensional Scaling

Exploring Latent States of Problem-Solving Competence Using Hidden Markov Model on Process Data

Multidimensional Latent Cognitive Diagnosis Based on Rough Concept Analysis and Item Response Theory

Assessing Preknowledge Cheating via Innovative Measures: A Multiple-Group Analysis of Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

A Fused Latent and Graphical Model for Multivariate Binary Data

3D computational modeling and perceptual analysis of kinetic depth effects

Latent Class Analysis of Recurrent Events in Problem-Solving Items

From Complexity to Parsimony: Integrating Latent Class Analysis to Uncover Multimodal Learning Patterns in Collaborative Learning

Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders

Unraveling Temporal Dynamics of Multidimensional Statistical Learning in Implicit and Explicit Systems: An X‐Way Hypothesis

New Paradigm of Identifiable General-response Cognitive Diagnostic Models: Beyond Categorical Data

Modeling eye movement patterns to characterize perceptual skill in image-based diagnostic reasoning processes

Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models

Latent class analysis for multi-layer categorical data

Multimodal Data Fusion to Detect Preknowledge Test-Taking Behavior Using Machine Learning