Abstract:Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients’ data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients’ data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration. Graphical abstract The workflow of proposed multi-modal image registration framework

Method and Application on the Localization of Feature-Point Pairs in Multi-Modal Medical Images

Local Registration of Medical Images Using Feature Point and Intensity Information

Registration of Medical Images Using Points and Curves

Medical Image Elastic Registration Based on Contour and Feature Points

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Manifold-based Feature Point Matching for Multi-Modal Image Registration.

Multimodal Medical Image Registration Via Common Representations Learning and Differentiable Geometric Constraints

Multi-Modal Image Registration Based on Multi-Feature Mutual Information

An Unsupervised Convolution Neural Network for Deformable Registration of Mono/Multi-Modality Medical Images

Image Synthesis-Based Multi-Modal Image Registration Framework by Using Deep Fully Convolutional Networks

Self-Supervised Keypoint Detection and Cross-Fusion Matching Networks for Multimodal Remote Sensing Image Registration

Registration of Multi-Modal Images under a Complex Background Combining Multiscale Features Extraction and Semantic Segmentation.

Multimodal registration network with multi-scale feature-crossing

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Image Registration Combining Cross-Scale Point Matching and Multi-Scale Feature Fusion

Multi-modal Medical Image Registration Based on Phase Congruency and Quantitative-Qualitative Mutual Information

Multi-modality Medical Image Registration Using Support Vector Machines.

Transformer-Based Local Feature Matching for Multimodal Image Registration

Localization: A Missing Link in the Pipeline of Object Matching and Registration

Non-Rigid Point Set Registration Using Multi-Feature and Gaussian Mixture Model

Multi-scale Feature Fusion Convolutional Neural Network for Multi-Modal Medical Image Fusion.