Abstract:Motivation: Recent brain mapping efforts are producing large-scale whole-brain images using different imaging modalities. Accurate alignment and delineation of anatomical structures in these images are essential for numerous studies. These requirements are typically modeled as two distinct tasks: Registration and segmentation. However, prevailing methods, fail to fully explore and utilize the inherent correlation and complementarity between the two tasks. Furthermore, variations in brain anatomy, brightness and texture pose another formidable challenge in designing multi-modal similarity metrics. A high-throughput approach capable of overcomingthe bottleneck of multi-modal similarity metric design, while effective leveraging the highly correlated and complementary nature of two tasks is highly desirable. Results: We introduce a deep learning framework for joint registration and segmentation of multi-modal brain images. Under this framework, registration and segmentation tasks are deeply coupled and collaborated at two hierarchical layers. In the inner layer, we establish a strong feature-level coupling between the two tasks by learning a unified common latent feature representation. In the outer layer, we introduce a mutually supervised dual-branch network to decouple latent features and facilitate task-level collaboration between registration and segmentation. Since the latent features we designed are also modality-independent, the bottleneck of designing multi-modal similarity metric is essentially addressed. Another merit offered by this framework is the interpretability of latent features, which allows intuitive manipulation of feature learning, thereby further enhancing network training efficiency and the performance of both tasks. Extensive experiments conducted on both multi-modal and mono-modal datasets of mouse and human brains demonstrate the superiority of our method. Availability: The code is available at https://github.com/tingtingup/DCRS. Supplementary information: Supplementary data are available at Bioinformaticsonline.

DD_RoTIR: Dual-Domain Image Registration via Image Translation and Hierarchical Feature-matching

RoTIR: Rotation-Equivariant Network and Transformers for Fish Scale Image Registration

TransDIR: Deformable imaging registration network based on transformer to improve the feature extraction ability

Interpretable Multi-Modal Image Registration Network Based on Disentangled Convolutional Sparse Coding

multiGradICON: A Foundation Model for Multimodal Medical Image Registration

Framework for Deep Learning Based Multi-Modality Image Registration of Snapshot and Pathology Images

UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration

Diffusion tensor driven image registration: a deep learning approach

End-to-end affine registration framework for histopathological images with weak annotations

RoReg: Pairwise Point Cloud Registration with Oriented Descriptors and Local Rotations

A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints

TransMatch: A Transformer-Based Multilevel Dual-Stream Feature Matching Network for Unsupervised Deformable Image Registration

Learning Dual Transformer Network for Diffeomorphic Registration

Transformer-Based Local Feature Matching for Multimodal Image Registration

DMDIT: Diverse multi-domain image-to-image translation

Synergistic registration of CT-MRI brain images and retinal images: A novel approach leveraging reinforcement learning and modified artificial rabbit optimization

Multi-modal Image Registration for Correlative Microscopy

DRFD-Net: Using Dual Receptive Field Descriptors for Multitemporal Optical Remote Sensing Image Registration

OTMorph: Unsupervised Multi-domain Abdominal Medical Image Registration Using Neural Optimal Transport

Deep coupled registration and segmentation of multimodal whole-brain images

General cross-modality registration framework for visible and infrared UAV target image registration