Abstract:Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients’ data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients’ data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework

A Deep Metric for Multimodal Registration

Multimodal Medical Image Registration Via Common Representations Learning and Differentiable Geometric Constraints

DISA: DIfferentiable Similarity Approximation for Universal Multimodal Registration

Regional Weighted Mutual Information For Multimodal Rigid Registration

Learning a Metric for Multimodal Medical Image Registration without Supervision Based on Cycle Constraints

Hessian-based Similarity Metric for Multimodal Medical Image Registration

Weakly-supervised convolutional neural networks for multimodal image registration

Deep learning-based affine medical image registration for multimodal minimal-invasive image-guided interventions – A comparative study on generalizability

Learning Deep Similarity Metric for 3D MR-TRUS Registration

Learning deep similarity metric for 3D MR–TRUS image registration

Metric Learning for Image Registration

Unsupervised deep learning registration model for multimodal brain images

Deep learning-based 3D brain multimodal medical image registration

multiGradICON: A Foundation Model for Multimodal Medical Image Registration

Fast and Robust Multimodal Image Registration Using A Local Derivative Pattern

Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity

Mutual information neural estimation for unsupervised multi-modal registration of brain images

Image Synthesis-Based Multi-Modal Image Registration Framework by Using Deep Fully Convolutional Networks

MvMM-RegNet: A new image registration framework based on multivariate mixture model and neural network estimation

Multimodality image registration by maximization of mutual information

End-to-end multimodal image registration via reinforcement learning