Abstract:The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

Real-Time Tracking Of Corneal Contour In Dalk Surgical Navigation Using Deep Neural Networks

Real-time segmentation and tracking of excised corneal contour by deep neural networks for DALK surgical navigation

Augmented Reality Surgical Navigation System Integrated with Deep Learning

ST-ITEF: Spatio-Temporal Intraoperative Task Estimating Framework to recognize surgical phase and predict instrument path based on multi-object tracking in keratoplasty

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Towards Deep Learning Guided Autonomous Eye Surgery Using Microscope and iOCT Images

Video-based Soft Tissue Deformation Tracking for Laparoscopic Augmented Reality-based Navigation in Kidney Surgery

Efficient Spatiotemporal Learning of Microscopic Video for Augmented Reality-Guided Phacoemulsification Cataract Surgery

Surgical Navigation System Based on the Visual Object Tracking Algorithm

Learning to Segment Corneal Tissue Interfaces in OCT Images

Long term and robust 6DoF motion tracking for highly dynamic stereo endoscopy videos

Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction

Augmented reality based surgery navigation system

Development of a Surgical Navigation System Based on Augmented Reality Using an Optical See-Through Head-Mounted Display.

Unsupervised corneal contour extraction algorithm with shared model for dynamic deformation videos: improving accuracy and noise resistance

Deep learning network to correct axial and coronal eye motion in 3D OCT retinal imaging

Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing

Reimagining partial thickness keratoplasty: An eye mountable robot for autonomous big bubble needle insertion

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Augmented Reality for Depth Cues in Monocular Minimally Invasive Surgery

Augmented Reality Navigation with Real-Time Tracking for Facial Repair Surgery