Abstract:Breast cancer is a significant global public health concern, with various treatment options available based on tumor characteristics. Pathological examination of excision specimens after surgery provides essential information for treatment decisions. However, the manual selection of representative sections for histological examination is laborious and subjective, leading to potential sampling errors and variability, especially in carcinomas that have been previously treated with chemotherapy. Furthermore, the accurate identification of residual tumors presents significant challenges, emphasizing the need for systematic or assisted methods to address this issue. In order to enable the development of deep-learning algorithms for automated cancer detection on radiology images, it is crucial to perform radiology-pathology registration, which ensures the generation of accurately labeled ground truth data. The alignment of radiology and histopathology images plays a critical role in establishing reliable cancer labels for training deep-learning algorithms on radiology images. However, aligning these images is challenging due to their content and resolution differences, tissue deformation, artifacts, and imprecise correspondence. We present a novel deep learning-based pipeline for the affine registration of faxitron images, the x-ray representations of macrosections of ex-vivo breast tissue, and their corresponding histopathology images of tissue segments. The proposed model combines convolutional neural networks and vision transformers, allowing it to effectively capture both local and global information from the entire tissue macrosection as well as its segments. This integrated approach enables simultaneous registration and stitching of image segments, facilitating segment-to-macrosection registration through a puzzling-based mechanism. To address the limitations of multi-modal ground truth data, we tackle the problem by training the model using synthetic mono-modal data in a weakly supervised manner. The trained model demonstrated successful performance in multi-modal registration, yielding registration results with an average landmark error of 1.51 mm (±2.40), and stitching distance of 1.15 mm (±0.94). The results indicate that the model performs significantly better than existing baselines, including both deep learning-based and iterative models, and it is also approximately 200 times faster than the iterative approach. This work bridges the gap in the current research and clinical workflow and has the potential to improve efficiency and accuracy in breast cancer evaluation and streamline pathology workflow.

Improving Algorithm for the Alignment of Consecutive, Whole-Slide, Immunohistochemical Section Images

Accurate and Robust Alignment of Differently Stained Histologic Images Based on Greedy Diffeomorphic Registration

Accurate and Robust Alignment of Variable-stained Histologic Images Using a General-purpose Greedy Diffeomorphic Registration Tool

Accurately Matching Serial Whole Slide Images for Tumor Heterogeneity Analysis

Comparison of consecutive and restained sections for image registration in histopathology

Comparison of Consecutive and Re-stained Sections for Image Registration in Histopathology

Point-Based Registration for Multi-stained Histology Images

Deep learning based registration of serial whole-slide histopathology images in different stains

Multimodal Alignment of Histopathological Images Using Cell Segmentation and Point Set Matching for Integrative Cancer Analysis

Whole slide image registration for the study of tumor heterogeneity

MSGM: an Advanced Deep Multi-Size Guiding Matching Network for Whole Slide Histopathology Images Addressing Staining Variation and Low Visibility Challenges

Regional Registration of Whole Slide Image Stacks Containing Highly Deformed Artefacts

Region-aware Registration for Multi-stained Histology Images.

A hierarchical and multi-view registration of serial histopathological images

PViT-AIR: Puzzling vision transformer-based affine image registration for multi histopathology and faxitron images of breast tissue

Automated quantitative analysis of Ki-67 staining and HE images recognition and registration based on whole tissue sections in breast carcinoma

A Hybrid Deep Feature-Based Deformable Image Registration Method for Pathology Images

Nuclei-Location Based Point Set Registration of Multi-Stained Whole Slide Images

Robust, fast and accurate: a 3-step method for automatic histological image registration

An Open-Source Whole Slide Image Registration Workflow at Cellular Precision Using Fiji, QuPath and Elastix

Virtual alignment of pathology image series for multi-gigapixel whole slide images