Abstract:Registration of pre-operative 3D volumes and intra-operative 2D images is critical for neurological interventions. In various 2D/3D registration tasks, deep learning-based approaches have become popular and achieved tremendous success. However, due to vast space of transformation parameters, estimation errors are significant in these approaches. To tackle above issues, a novel learning-based framework for 2D/3D registration is proposed, consisting of CNN regression and centroid alignment. The former introduces a residual regression network (Res-RegNet) to preliminarily estimate transformation parameters. To further reduce estimation errors, the latter utilizes target vessel centroids to refine projected images. The proposed framework is individually trained and evaluated on three patients, reaching mean Dice of 76.69%, 78.51%, and 85.39%, respectively, all outperforming baseline methods. Extensive ablation studies demonstrate centroid alignment can significantly improve registration performance. As a normalization layer in Res-RegNet, SPADE can modulate activations using binarized inputs through a spatially-adaptive, learned transformation. Semantic information of inputs is preserved to learn better representations for parameter estimation. Moreover, the inference rate of our framework is about 21 FPS combined with the state-of-the-art segmentation model, significantly surpassing real-time requirements (6 $\sim$ 12 FPS) in clinical practice. These promising results indicate the potential of the framework to facilitate various 2D/3D registration tasks. Note to Practitioners—This paper was motivated by the problem of image-guided neurological interventions. Existing 2D/3D registration methods suffer from 1) long iteration times, which are difficult to meet real-time clinical necessities, or 2) significant parameter estimation errors, leading to poor registration accuracies. Therefore, this paper suggests a new registration framework, combining with CNN regression to give predictions of transformation parameters via a single forward propagation, and centroid alignment to reduce estimation errors by translation transformation. The framework is trained and tested on three patients separately and achieves state-of-the-art performance, demonstrating its superiority. Furthermore, the proposed framework is a learning-based method that is adaptable to various image modalities. Therefore, it has latent capacities to be integrated into surgical navigation systems.

Real-time 6DoF pose recovery from X-ray images using library-based DRR and hybrid optimization

Isocentric fixed angle irradiation-based DRR: a novel approach to enhance x-ray and CT image registration

The Impact of Loss Functions and Scene Representations for 3D/2D Registration on Single-view Fluoroscopic X-ray Pose Estimation

DSR: Direct Simultaneous Registration for Multiple 3D Images

A novel 2D/3D hierarchical registration framework via principal-directional Fourier transform operator

Personalized X-ray Reconstruction of the Proximal Femur Via a New Control Point-based 2D-3D Registration and Residual Complexity Minimization.

Real-time CBCT Imaging and Motion Tracking via a Single Arbitrarily-angled X-ray Projection by a Joint Dynamic Reconstruction and Motion Estimation (DREME) Framework

Robust Orthogonal-View 2-D/3-D Rigid Registration for Minimally Invasive Surgery

Hardware Accelerated Ray Cast of Volume Data and Volume Gradient for an Optimized Splines-Based Multi-Resolution 2D-3D Registration

Registration of On-Board X-ray Images with 4DCT: A Proposed Method of Phase and Setup Verification for Gated Radiotherapy

Registration of 3D trans-esophageal echocardiography to X-ray fluoroscopy using image-based probe tracking

DRR acceleration using inexpensive GPUs for model-image registration based joint kinematic measurements

A fast DRR-based image regestrition algorithm for calculation of setup error in radiotherapy

Real-Time 2D/3D Registration via CNN Regression and Centroid Alignment

High-accuracy and Real-Time 3D Positioning, Tracking System for Medical Imaging Applications Based on 3D Digital Image Correlation

Machine learning for automated and real-time two-dimensional to three-dimensional registration of the spine using a single radiograph

CT2X-IRA: CT to x-ray image registration agent using domain-cross multi-scale-stride deep reinforcement learning

Automatic 3D-to-2d Registration for Ct and Dual-Energy Digital Radiography for Calcification Detection

RT-SRTS: Angle-Agnostic Real-Time Simultaneous 3D Reconstruction and Tumor Segmentation from Single X-Ray Projection

Fully automated 2D-3D registration and verification