Abstract:Motivation: Recent brain mapping efforts are producing large-scale whole-brain images using different imaging modalities. Accurate alignment and delineation of anatomical structures in these images are essential for numerous studies. These requirements are typically modeled as two distinct tasks: Registration and segmentation. However, prevailing methods, fail to fully explore and utilize the inherent correlation and complementarity between the two tasks. Furthermore, variations in brain anatomy, brightness and texture pose another formidable challenge in designing multi-modal similarity metrics. A high-throughput approach capable of overcomingthe bottleneck of multi-modal similarity metric design, while effective leveraging the highly correlated and complementary nature of two tasks is highly desirable. Results: We introduce a deep learning framework for joint registration and segmentation of multi-modal brain images. Under this framework, registration and segmentation tasks are deeply coupled and collaborated at two hierarchical layers. In the inner layer, we establish a strong feature-level coupling between the two tasks by learning a unified common latent feature representation. In the outer layer, we introduce a mutually supervised dual-branch network to decouple latent features and facilitate task-level collaboration between registration and segmentation. Since the latent features we designed are also modality-independent, the bottleneck of designing multi-modal similarity metric is essentially addressed. Another merit offered by this framework is the interpretability of latent features, which allows intuitive manipulation of feature learning, thereby further enhancing network training efficiency and the performance of both tasks. Extensive experiments conducted on both multi-modal and mono-modal datasets of mouse and human brains demonstrate the superiority of our method. Availability: The code is available at https://github.com/tingtingup/DCRS. Supplementary information: Supplementary data are available at Bioinformaticsonline.

Bidirectional Mapping with Contrastive Learning on Multimodal Neuroimaging Data

Multimodal Brain Disease Classification with Functional Interaction Learning from Single fMRI Volume

Multimodal Contrastive Learning for Brain-Machine Fusion: from Brain-in-the-loop Modeling to Brain-out-of-the-loop Application

Mapping Multi-Modal Brain Connectome for Brain Disorder Diagnosis Via Cross-Modal Mutual Learning.

Incomplete Learning of Multi-Modal Connectome for Brain Disorder Diagnosis Via Modal-Mixup and Deep Supervision.

Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment

Deep coupled registration and segmentation of multimodal whole-brain images

Multi-instance Discriminative Contrastive Learning for Brain Image Representation

Multimodal functional deep learning for multiomics data

Bidirectional Mapping of Brain MRI and PET with 3D Reversible GAN for the Diagnosis of Alzheimer's Disease.

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

On self-supervised multi-modal representation learning: An application to Alzheimer's disease

Functional2Structural: Cross-Modality Brain Networks Representation Learning

Brain-ID: Learning Contrast-agnostic Anatomical Representations for Brain Imaging

Bidirectional Mapping of Brain MRI and PET With 3D Reversible GAN for the Diagnosis of Alzheimer’s Disease

Multimodal identification of the mouse brain using simultaneous Ca imaging and fMRI

Deep sr-DDL: Deep structurally regularized dynamic dictionary learning to integrate multimodal and dynamic functional connectomics data for multidimensional clinical characterizations

Contrastive Learning on Multimodal Analysis of Electronic Health Records

In vivo cell-type and brain region classification via multimodal contrastive learning

Mapping EEG Signals to Visual Stimuli: A Deep Learning Approach to Match Vs. Mismatch Classification

Inter-modality Relationship Constrained Multi-Task Feature Selection for AD/MCI Classification.