Abstract:Co-examination of second-harmonic generation (SHG) and bright-field (BF) microscopy enables the differentiation of tissue components and collagen fibers, aiding the analysis of human breast and pancreatic cancer tissues. However, large discrepancies between SHG and BF images pose challenges for current learning-based registration models in aligning SHG to BF. In this paper, we propose a novel multi-modal registration framework that employs fidelity-imposed displacement editing to address these challenges. The framework integrates batch-wise contrastive learning, feature-based pre-alignment, and instance-level optimization. Experimental results from the Learn2Reg COMULISglobe SHG-BF Challenge validate the effectiveness of our method, securing the 1st place on the online leaderboard.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in multi - modal medical image registration, how to effectively align second - harmonic generation (SHG) microscopic images and bright - field (BF) microscopic images. Specifically, SHG images emphasize collagen fibers, while BF images highlight stained tissue components. There are significant visual differences between these two modalities, which makes it difficult for existing learning - based registration models to accurately align these images. ### Problem Background 1. **Modal Differences**: SHG images and BF images are very different in imaging mechanisms and visual features. SHG images mainly capture collagen fibers, while BF images highlight various tissue components through H&E staining. 2. **Foreground - Background Imbalance**: The feature distribution in SHG images is sparse, resulting in a serious imbalance between the foreground and the background. This poses a challenge to deep - learning - based methods (such as convolutional neural networks, visual transformers, and keypoint - based methods). ### Solution To solve the above problems, the author proposes a new multi - modal registration framework. Its main contributions include: 1. **Batch - wise Noise - Contrastive Estimation Loss (B - NCE)**: The batch - wise noise - contrastive estimation loss is introduced to effectively capture the shared features between SHG and BF images. This loss function enhances the network's ability to capture cross - modal shared features by focusing on the position - level similarity in the batch dimension. 2. **Feature - Based Pre - Alignment and Instance - Level Optimization**: First, pre - alignment is performed using descriptor matching, and then instance - level optimization is carried out to refine the registration results. This step calculates the initial affine transformation through feature matching and further optimizes it to improve the registration accuracy. 3. **Flexible Similarity Metric Transformation**: Combining local normalized cross - correlation (LNCC) and cross - mutual information function (CMIF) as similarity metrics to balance global and local alignment, so as to better deal with the modal gap and foreground - background imbalance problems. ### Experimental Results The experimental results show that this method achieved the first place on the online leaderboard of the Learn2Reg COMULISglobe SHG - BF Challenge, verifying its effectiveness and robustness. Both quantitative and qualitative results show that this method has a significant improvement in registration accuracy and stability. ### Formula Representation The formulas involved in the paper are as follows: - **Cosine Similarity**: \[ s(x_{i,j}, y_{i,j})=\frac{x_{i,j}^{\top}y_{i,j}}{\left\|x_{i,j}\right\|\left\|y_{i,j}\right\|} \] - **B - NCE Loss**: \[ L_{i,j}^{\text{B - NCE}} =-\log\left(\frac{\exp(s(x_{i,j}, y_{i,j}) / \tau)}{\sum_{k = 1}^{N}\exp(s(x_{i,j}, z_k)/\tau)}\right) \] where \(z_k\) includes positive and negative samples, and \(\tau\) is the temperature parameter. - **Total B - NCE Loss**: \[ L^{\text{B - NCE}}=\frac{1}{H_pW_p}\sum_{i = 1}^{H_p}\sum_{j = 1}^{W_p}L_{i,j}^{\text{B - NCE}} \] Through these methods and techniques, the author has successfully solved the challenges in SHG and BF image registration and provided more accurate and robust registration performance.

Fidelity-Imposed Displacement Editing for the Learn2Reg 2024 SHG-BF Challenge

Automatic Registration of SHG and H&E Images with Feature-based Initial Alignment and Intensity-based Instance Optimization: Contribution to the COMULIS Challenge

Intensity-based registration of bright-field and second-harmonic generation images of histopathology tissue sections

Multimodal Medical Image Registration Via Common Representations Learning and Differentiable Geometric Constraints

Color Fundus Photograph Registration Based on Feature and Intensity for Longitudinal Evaluation of Diabetic Retinopathy Progression

Unsupervised Multimodal 3D Medical Image Registration with Multilevel Correlation Balanced Optimization

Homographic Patch Feature Transform: A Robustness Registration for Gastroscopic Surgery.

Boosting Medical Image Synthesis via Registration-guided Consistency and Disentanglement Learning

Multimodal Affine Registration for ICGA and MCSL Fundus Images of High Myopia

Anatomically Constrained and Attention-Guided Deep Feature Fusion for Joint Segmentation and Deformable Medical Image Registration.

3D Biological/Biomedical Image Registration with enhanced Feature Extraction and Outlier Detection

LDVoxelMorph: A Precise Loss Function and Cascaded Architecture for Unsupervised Diffeomorphic Large Displacement Registration.

MA-MFIF: When misaligned multi-focus Image fusion meets deep homography estimation

Accurate and Robust Alignment of Differently Stained Histologic Images Based on Greedy Diffeomorphic Registration

A Coarse-to-Fine Deformable Transformation Framework for Unsupervised Multi-Contrast MR Image Registration with Dual Consistency Constraint

A coarse-to-fine framework for unsupervised multi-contrast MR image deformable registration with dual consistency constraint

A Hybrid Deep Feature-Based Deformable Image Registration Method for Pathology Images

Registration of in vivo fluorescence endomicroscopy images based on feature detection.

Multilevel network for large deformation image registration based on feature consistency and flow normalization

Progressive Retinal Image Registration via Global and Local Deformable Transformations

Deep Complementary Joint Model for Complex Scene Registration and Few-Shot Segmentation on Medical Images