Abstract:Registration of pre-operative 3D volumes and intra-operative 2D images is critical for neurological interventions. In various 2D/3D registration tasks, deep learning-based approaches have become popular and achieved tremendous success. However, due to vast space of transformation parameters, estimation errors are significant in these approaches. To tackle above issues, a novel learning-based framework for 2D/3D registration is proposed, consisting of CNN regression and centroid alignment. The former introduces a residual regression network (Res-RegNet) to preliminarily estimate transformation parameters. To further reduce estimation errors, the latter utilizes target vessel centroids to refine projected images. The proposed framework is individually trained and evaluated on three patients, reaching mean Dice of 76.69%, 78.51%, and 85.39%, respectively, all outperforming baseline methods. Extensive ablation studies demonstrate centroid alignment can significantly improve registration performance. As a normalization layer in Res-RegNet, SPADE can modulate activations using binarized inputs through a spatially-adaptive, learned transformation. Semantic information of inputs is preserved to learn better representations for parameter estimation. Moreover, the inference rate of our framework is about 21 FPS combined with the state-of-the-art segmentation model, significantly surpassing real-time requirements (6 $\sim$ 12 FPS) in clinical practice. These promising results indicate the potential of the framework to facilitate various 2D/3D registration tasks. Note to Practitioners—This paper was motivated by the problem of image-guided neurological interventions. Existing 2D/3D registration methods suffer from 1) long iteration times, which are difficult to meet real-time clinical necessities, or 2) significant parameter estimation errors, leading to poor registration accuracies. Therefore, this paper suggests a new registration framework, combining with CNN regression to give predictions of transformation parameters via a single forward propagation, and centroid alignment to reduce estimation errors by translation transformation. The framework is trained and tested on three patients separately and achieves state-of-the-art performance, demonstrating its superiority. Furthermore, the proposed framework is a learning-based method that is adaptable to various image modalities. Therefore, it has latent capacities to be integrated into surgical navigation systems.

Fully convolutional network-based registration for augmented assembly systems

Integrated Registration and Occlusion Handling Based on Deep Learning for Augmented Reality Assisted Assembly Instruction

FDN: Feature Decoupling Network for Head Pose Estimation.

A fast object registration method for augmented reality assembly with simultaneous determination of multiple 2D-3D correspondences

A Skeleton-Based Assembly Action Recognition Method with Feature Fusion for Human-Robot Collaborative Assembly

Deep learning-based augmented reality work instruction assistance system for complex manual assembly

Local geometric edge features based registration for textureless object in augmented reality assisted assembly

Monitoring of Assembly Process Using Deep Learning Technology.

DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration

A 3D Tracking and Registration Method Based on Point Cloud and Visual Features for Augmented Reality Aided Assembly System

Non-rigid image registration using fully convolutional networks with deep self-supervision

Assembly Monitoring Using Semantic Segmentation Network Based on Multiscale Feature Maps and Trainable Guided Filter

A rapid oriented detection method of virtual components for augmented assembly

SEN-FCB: an Unsupervised Twinning Neural Network for Image Registration

AN AUTOMATIC INTERACTION METHOD USING PART RECOGNITION BASED ON DEEP NETWORK FOR AUGMENTED REALITY ASSEMBLY GUIDANCE

MAFNet: A two-stage multiple attention fusion network for partial-to-partial point cloud registration

A Novel Method of Multitarget Augmented Reality Assembly Result Inspection for Large Complex Scenes

Improved Deep Learning-Based Efficientpose Algorithm for Egocentric Marker-Less Tool and Hand Pose Estimation in Manual Assembly.

Real-Time 2D/3D Registration via CNN Regression and Centroid Alignment

Non-rigid image registration using self-supervised fully convolutional networks without training data