Abstract:The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

Real-time Surgical Tool Tracking and Pose Estimation Using a Hybrid Cylindrical Marker.

Real-time 3D Tracking of Articulated Tools for Robotic Surgery

Markerless Inside-Out Tracking for Interventional Applications

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Spatial Position Measurement System for Surgical Navigation Using 3-D Image Marker-Based Tracking Tools with Compact Volume

Design of the Image-Guided Biopsy Marking System for Gastroscopy

A Markerless 3D Tracking Framework for Continuum Surgical Tools Using a Surgical Tool Partial Pose Estimation Network Based on Domain Randomization

Performance Enhanced Ultrasound Probe Tracking With a Hemispherical Marker Rigid Body

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Real-Time Marker-Free Patient Registration and Image-Based Navigation Using Stereovision for Dental Surgery.

Visualization and Real-time Tracking Technologies of the Probe Used in Surgical Navigation Based on Electromagnetic Positioning

Wearable Surgical Optical Tracking System Based on Multi-Modular Sensor Fusion

Multisource Differential Fusion Driven Monocular Endoscope Hybrid 3-D Tracking for Advanced Endoscopic Navigation Surgery

Vision-Based Markerless Tracking for Continuum Surgical Instruments in Robot-Assisted Minimally Invasive Surgery

TAToo: vision-based joint tracking of anatomy and tool for skull-base surgery

Real‐time Motion Tracking in Image‐guided Oral Implantology

Towards Tracking by 2D-Target Registration for Surgical Optical Tracking System

Ultrasound Image-Guided Mapping of Endoscopic Views on a 3D Placenta Model: A Tracker-Less Approach.

Surgical Navigation System Based on the Visual Object Tracking Algorithm

3D Tracking and Positioning of Surgical Instruments in Virtual Surgery Simulation

An intelligent tracking system for surgical instruments in complex surgical environment