Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching

Stefan Brandstätter,Philipp Seeböck,Christoph Fürböck,Svitlana Pochepnia,Helmut Prosch,Georg Langs
DOI: https://doi.org/10.1007/978-3-031-73480-9_22
2024-10-24
Abstract:2D to 3D registration is essential in tasks such as diagnosis, surgical navigation, environmental understanding, navigation in robotics, autonomous systems, or augmented reality. In medical imaging, the aim is often to place a 2D image in a 3D volumetric observation to w. Current approaches for rigid single slice in volume registration are limited by requirements such as pose initialization, stacks of adjacent slices, or reliable anatomical landmarks. Here, we propose a self-supervised 2D/3D registration approach to match a single 2D slice to the corresponding 3D volume. The method works in data without anatomical priors such as images of tumors. It addresses the dimensionality disparity and establishes correspondences between 2D in-plane and 3D out-of-plane rotation-equivariant features by using group equivariant CNNs. These rotation-equivariant features are extracted from the 2D query slice and aligned with their 3D counterparts. Results demonstrate the robustness of the proposed slice-in-volume registration on the NSCLC-Radiomics CT and KIRBY21 MRI datasets, attaining an absolute median angle error of less than 2 degrees and a mean-matching feature accuracy of 89% at a tolerance of 3 pixels.
Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
### What problem does this paper attempt to solve? This paper aims to solve the problem of accurately aligning a single 2D image slice into a 3D volumetric image, namely the so - called "Slice - in - Volume" registration problem. Specifically, the authors propose a self - supervised 2D/3D registration method. Through rotation - equivariant feature matching, it can achieve precise positioning of 2D image slices in 3D volumes without anatomical prior knowledge. #### Main problems and challenges: 1. **Initialization requirements**: Existing methods usually require initial pose estimation or adjacent slice stacks, which limits their application range. 2. **Dependence on anatomical landmarks**: Many methods rely on reliable anatomical landmarks, which become unreliable when dealing with irregular structures (such as tumors). 3. **Dimension difference**: A 2D slice contains only a small part of the information of a 3D volume. How to establish the correspondence between in - plane rotation and out - of - plane rotation in the 2D plane is a challenge. 4. **High computational cost**: Traditional optimization methods require a large number of optimization steps, resulting in high computational costs and being sensitive to initial values. #### Solutions: The authors propose a method named SLIV - Reg, which uses group rotation - equivariant convolutional neural networks (group equivariant CNNs) to extract rotation - equivariant features, thus solving the above problems. Specifically, SLIV - Reg is implemented in the following ways: - **Feature extraction**: Use a rotation - equivariant ResNet - 18 model to extract feature vectors from 2D query slices and 3D search volumes. - **Candidate point selection**: Use the Canny edge detector to identify candidate points to reduce computational costs. - **Multi - directional sampling**: In order to deal with out - of - plane rotation, efficiently sample 2D features in different directions from the 3D volume. - **Feature matching**: Calculate the L2 distance between feature vectors to find the closest matching points. - **Pose prediction**: Use the RANSAC algorithm to predict the position of the 2D slice in the 3D volume according to the matching points. #### Experimental results: The experimental results show that SLIV - Reg performs excellently on the NSCLC - Radiomics CT and Kirby21 MRI datasets, with an absolute median angular error of less than 2 degrees and a feature - matching accuracy rate of 89% (under a 3 - pixel tolerance). This indicates that this method is robust and accurate in dealing with complex medical image data. Through this method, SLIV - Reg overcomes the limitations of existing methods and can achieve reliable registration of 2D slices in 3D volumes without the need for initialization, iterative refinement, anatomical prior knowledge, etc.