Abstract:Motivation: Recent brain mapping efforts are producing large-scale whole-brain images using different imaging modalities. Accurate alignment and delineation of anatomical structures in these images are essential for numerous studies. These requirements are typically modeled as two distinct tasks: Registration and segmentation. However, prevailing methods, fail to fully explore and utilize the inherent correlation and complementarity between the two tasks. Furthermore, variations in brain anatomy, brightness and texture pose another formidable challenge in designing multi-modal similarity metrics. A high-throughput approach capable of overcomingthe bottleneck of multi-modal similarity metric design, while effective leveraging the highly correlated and complementary nature of two tasks is highly desirable. Results: We introduce a deep learning framework for joint registration and segmentation of multi-modal brain images. Under this framework, registration and segmentation tasks are deeply coupled and collaborated at two hierarchical layers. In the inner layer, we establish a strong feature-level coupling between the two tasks by learning a unified common latent feature representation. In the outer layer, we introduce a mutually supervised dual-branch network to decouple latent features and facilitate task-level collaboration between registration and segmentation. Since the latent features we designed are also modality-independent, the bottleneck of designing multi-modal similarity metric is essentially addressed. Another merit offered by this framework is the interpretability of latent features, which allows intuitive manipulation of feature learning, thereby further enhancing network training efficiency and the performance of both tasks. Extensive experiments conducted on both multi-modal and mono-modal datasets of mouse and human brains demonstrate the superiority of our method. Availability: The code is available at https://github.com/tingtingup/DCRS. Supplementary information: Supplementary data are available at Bioinformaticsonline.

Consistent cross-modal identification of cortical neurons with coupled autoencoders

A coupled autoencoder approach for multi-modal analysis of cell types

In vivo cell-type and brain region classification via multimodal contrastive learning

Multimodal identification of the mouse brain using simultaneous Ca imaging and fMRI

Multimodal identification of the mouse brain using simultaneous Ca 2+ imaging and fMRI

MultiCPA: Multimodal Compositional Perturbation Autoencoder

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Graph-based cell pattern recognition for merging the multi-modal optical microscopic image of neurons

Combining biophysical modeling and deep learning for multielectrode array neuron localization and classification

Integrating single-cell multimodal epigenomic data using 1D-convolutional neural networks

PhysMAP - interpretable neuronal cell type identification using multi-modal analysis of electrophysiological data

Reproducibility of predictive networks for mouse visual cortex

Cross-modal autoencoder framework learns holistic representations of cardiovascular state

Neuromorphic Networks as Revealed by Features Similarity

Neurons Identification of Single-Photon Wide-Field Calcium Fluorescent Imaging Data

Multi-modal Gaussian Process Variational Autoencoders for Neural and Behavioral Data

End-to-end multimodal deep learning for real-time decoding of months-long neural activity from the same cells

Deep coupled registration and segmentation of multimodal whole-brain images

Reconstruct a Connectome of Single Neurons in Mouse Brains by Cross-Validating Multi-Scale Multi-Modality Data

Self-Attention-Based Contextual Modulation Improves Neural System Identification

Mapping Multi-Modal Brain Connectome for Brain Disorder Diagnosis Via Cross-Modal Mutual Learning.