Abstract:Registration of pre-operative 3D volumes and intra-operative 2D images is critical for neurological interventions. In various 2D/3D registration tasks, deep learning-based approaches have become popular and achieved tremendous success. However, due to vast space of transformation parameters, estimation errors are significant in these approaches. To tackle above issues, a novel learning-based framework for 2D/3D registration is proposed, consisting of CNN regression and centroid alignment. The former introduces a residual regression network (Res-RegNet) to preliminarily estimate transformation parameters. To further reduce estimation errors, the latter utilizes target vessel centroids to refine projected images. The proposed framework is individually trained and evaluated on three patients, reaching mean Dice of 76.69%, 78.51%, and 85.39%, respectively, all outperforming baseline methods. Extensive ablation studies demonstrate centroid alignment can significantly improve registration performance. As a normalization layer in Res-RegNet, SPADE can modulate activations using binarized inputs through a spatially-adaptive, learned transformation. Semantic information of inputs is preserved to learn better representations for parameter estimation. Moreover, the inference rate of our framework is about 21 FPS combined with the state-of-the-art segmentation model, significantly surpassing real-time requirements (6 $\sim$ 12 FPS) in clinical practice. These promising results indicate the potential of the framework to facilitate various 2D/3D registration tasks. Note to Practitioners—This paper was motivated by the problem of image-guided neurological interventions. Existing 2D/3D registration methods suffer from 1) long iteration times, which are difficult to meet real-time clinical necessities, or 2) significant parameter estimation errors, leading to poor registration accuracies. Therefore, this paper suggests a new registration framework, combining with CNN regression to give predictions of transformation parameters via a single forward propagation, and centroid alignment to reduce estimation errors by translation transformation. The framework is trained and tested on three patients separately and achieves state-of-the-art performance, demonstrating its superiority. Furthermore, the proposed framework is a learning-based method that is adaptable to various image modalities. Therefore, it has latent capacities to be integrated into surgical navigation systems.

A deep unrolled neural network for real-time MRI-guided brain intervention

Deep Learning for Under-Sampled Non-Cartesian ASL MRI Reconstruction

Fast and Lightweight Network Improves Serial Brain Section Stitching

Parallel Imaging and Convolutional Neural Network Combined Fast MR Image Reconstruction: Applications in Low-Latency Accelerated Real-Time Imaging.

RT-DNAS: Real-Time Constrained Differentiable Neural Architecture Search for 3D Cardiac Cine MRI Segmentation

PILOT: Physics-Informed Learned Optimized Trajectories for Accelerated MRI

Deep learning enabled fast 3D brain MRI at 0.055 tesla

Super-resolution neural networks improve the spatiotemporal resolution of adaptive MRI-guided radiation therapy

Deep-learning based image reconstruction for MRI-guided near-infrared spectral tomography

Multi-modal networks for real-time monitoring of intracranial acoustic field during transcranial focused ultrasound therapy

Predicting 4D Liver MRI for MR-guided Interventions

Real-Time 2D/3D Registration via CNN Regression and Centroid Alignment

3D Ultrasonic Brain Imaging with Deep Learning Based on Fully Convolutional Networks

Real‐time Radial Reconstruction with domain transform manifold learning for MRI‐guided radiotherapy

Interactive Multi-Stage Robotic Positioner for Intra-Operative MRI-Guided Stereotactic Neurosurgery

Low-Rank and Framelet Based Sparsity Decomposition for Interventional MRI Reconstruction

On Real-time Image Reconstruction with Neural Networks for MRI-guided Radiotherapy

In-vivo verified anatomically aware deep learning for real-time electric field simulation

Deep learning-based automated scan plane positioning for brain magnetic resonance imaging

Instant tissue field and magnetic susceptibility mapping from MRI raw phase using Laplacian enhanced deep neural networks

Deep MRI Reconstruction: Unrolled Optimization Algorithms Meet Neural Networks