Abstract:Modern medical image registration approaches predict deformations using deep networks. These approaches achieve state-of-the-art (SOTA) registration accuracy and are generally fast. However, deep learning (DL) approaches are, in contrast to conventional non-deep-learning-based approaches, anatomy-specific. Recently, a universal deep registration approach, uniGradICON, has been proposed. However, uniGradICON focuses on monomodal image registration. In this work, we therefore develop multiGradICON as a first step towards universal *multimodal* medical image registration. Specifically, we show that 1) we can train a DL registration model that is suitable for monomodal *and* multimodal registration; 2) loss function randomization can increase multimodal registration accuracy; and 3) training a model with multimodal data helps multimodal generalization. Our code and the multiGradICON model are available at <a class="link-external link-https" href="https://github.com/uncbiag/uniGradICON" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to develop a foundational model for multimodal medical image registration called **multiGradICON** to address the limitations of existing deep learning methods in multimodal image registration. Specifically, the paper attempts to solve the following key issues: 1. **Generality of Multimodal Image Registration**: - Existing deep learning image registration methods typically focus on single-modal image registration and are trained for specific anatomical regions, which limits their generality. - Although **uniGradICON** is a general single-modal image registration method, it performs poorly in multimodal image registration. 2. **Accuracy of Multimodal Image Registration**: - Deep learning methods excel in single-modal image registration but face challenges in multimodal image registration, especially when there are significant modality differences. - The paper proposes to improve the accuracy of multimodal image registration through loss function randomization. 3. **Utilization of Multimodal Data**: - Existing methods usually do not include multimodal data during training, which limits the model's generalization ability in multimodal tasks. - The paper explores how to improve the model's multimodal generalization ability by introducing multimodal data during training. ### Main Contributions 1. **Extension of uniGradICON**: - Selecting similarity metrics suitable for multimodal image registration. - Incorporating multimodal registration tasks into the training process. - Exploring single-modal, multimodal, and randomized image similarity loss strategies. 2. **Demonstration of Multimodal Generalization Ability**: - Proving the limitations of uniGradICON in multimodal registration. - Showing that multiGradICON, trained with multimodal data, can achieve multimodal generalization while maintaining single-modal registration accuracy. 3. **Improvement in Registration Accuracy for Multi-parameter Datasets**: - Enhancing the registration accuracy of datasets containing multi-parameter data through loss function randomization (i.e., randomly selecting which modalities to compare in the loss), even when using scalar images during inference. ### Experimental Results - **Single-modal Datasets**: - On multiple single-modal datasets, the initial prediction performance of multiGradICON is slightly inferior to uniGradICON, but the performance gap significantly narrows after instance optimization. - On some newly introduced single-modal datasets, multiGradICON outperforms uniGradICON. - **Multimodal Datasets**: - multiGradICON performs excellently on multimodal datasets, especially when there are significant modality differences. - The loss function randomization strategy significantly improves the accuracy of multimodal image registration. ### Conclusion By developing the multiGradICON model, the paper successfully addresses the limitations of existing deep learning methods in multimodal medical image registration, improving the accuracy and generalization ability of multimodal image registration.

multiGradICON: A Foundation Model for Multimodal Medical Image Registration

uniGradICON: A Foundation Model for Medical Image Registration

Multimodal Medical Image Registration Via Common Representations Learning and Differentiable Geometric Constraints

$\texttt{GradICON}$: Approximate Diffeomorphisms via Gradient Inverse Consistency

A Deep Metric for Multimodal Registration

GAN Based Medical Image Registration

MICDIR: Multi-scale Inverse-consistent Deformable Image Registration using UNetMSS with Self-Constructing Graph Latent

An Unsupervised Convolution Neural Network for Deformable Registration of Mono/Multi-Modality Medical Images

Deep learning-based 3D brain multimodal medical image registration

Learn2Reg: Comprehensive Multi-Task Medical Image Registration Challenge, Dataset and Evaluation in the Era of Deep Learning

Explainable unsupervised multi-modal image registration using deep networks

Unsupervised Multimodal Image Registration with Adaptative Gradient Guidance

Image Synthesis-Based Multi-Modal Image Registration Framework by Using Deep Fully Convolutional Networks

GAN-DIRNet: A Novel Deformable Image Registration Approach for Multimodal Histological Images

HGCMorph: joint discontinuity-preserving and pose-learning via GNN and capsule networks for deformable medical images registration

Unsupervised MMRegNet Based on Spatially Encoded Gradient Information.

DELR-Net: a network for 3D multimodal medical image registration in more lightweight application scenarios

UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration

Multiscale unsupervised network for deformable image registration

MrRegNet: Multi-resolution Mask Guided Convolutional Neural Network for Medical Image Registration with Large Deformations

Networks for Joint Affine and Non-parametric Image Registration