Abstract:In recent years, deep learning approaches have gained significant attention in predicting brain disorders using neuroimaging data. However, conventional methods often rely on single-modality data and supervised models, which provide only a limited perspective of the intricacies of the highly complex brain. Moreover, the scarcity of accurate diagnostic labels in clinical settings hinders the applicability of the supervised models. To address these limitations, we propose a novel self-supervised framework for extracting multiple representations from multimodal neuroimaging data to enhance group inferences and enable analysis without resorting to labeled data during pre-training. Our approach leverages Deep InfoMax (DIM), a self-supervised methodology renowned for its efficacy in learning representations by estimating mutual information without the need for explicit labels. While DIM has shown promise in predicting brain disorders from single-modality MRI data, its potential for multimodal data remains untapped. This work extends DIM to multimodal neuroimaging data, allowing us to identify disorder-relevant brain regions and explore multimodal links. We present compelling evidence of the efficacy of our multimodal DIM analysis in uncovering disorder-relevant brain regions, including the hippocampus, caudate, insula, - and multimodal links with the thalamus, precuneus, and subthalamus hypothalamus. Our self-supervised representations demonstrate promising capabilities in predicting the presence of brain disorders across a spectrum of Alzheimer's phenotypes. Comparative evaluations against state-of-the-art unsupervised methods based on autoencoders, canonical correlation analysis, and supervised models highlight the superiority of our proposed method in achieving improved classification performance, capturing joint information, and interpretability capabilities. The computational efficiency of the decoder-free strategy enhances its practical utility, as it saves compute resources without compromising performance. This work offers a significant step forward in addressing the challenge of understanding multimodal links in complex brain disorders, with potential applications in neuroimaging research and clinical diagnosis.

Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data

The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning

BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data

Self-supervised pretraining improves the performance of classification of task functional magnetic resonance imaging

Self-supervised multimodal learning for group inferences from MRI data: Discovering disorder-relevant brain regions and multimodal links

Learning to Decode Cognitive States from Brain Images

Using deep generative models for simultaneous representational and predictive modeling of brain and behavior: A graded supervised-to-unsupervised modeling framework.

Challenges for cognitive decoding using deep learning methods

BrainMAE: A Region-aware Self-supervised Learning Framework for Brain Signals

Graph Self-supervised Learning with Application to Brain Networks Analysis

A Hybrid Spatiotemporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multilevel Core Functional Components in Decoding Multitask Fmri Signals.

A Hybrid Spatio-Temporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multi-Level Core Functional Components in Decoding Multi-Task fMRI Signals

BrainLM: A foundation model for brain activity recordings

Unsupervised decoding of long-term, naturalistic human neural recordings with automated video and audio annotations

Group-level Brain Decoding with Deep Learning

Comparing interpretation methods in mental state decoding analyses with deep learning models

Decoding the Large-Scale Structure of Brain Function by Classifying Mental States Across Individuals

Interpretable Self-Supervised Facial Micro-Expression Learning to Predict Cognitive State and Neurological Disorders

Toward a realistic model of speech processing in the brain with self-supervised learning

Self-supervised multimodal neuroimaging yields predictive representations for a spectrum of Alzheimer's phenotypes

Decoding Brain States From fMRI Signals by Using Unsupervised Domain Adaptation