Abstract:The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

TAToo: vision-based joint tracking of anatomy and tool for skull-base surgery

Real-time 3D Tracking of Articulated Tools for Robotic Surgery

SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Tracking Everything in Robotic-Assisted Surgery

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model.

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Real-time Surgical Tool Tracking and Pose Estimation Using a Hybrid Cylindrical Marker.

Plug-in for visualizing 3D tool tracking from videos of Minimally Invasive Surgeries

Robust 3D Motion Tracking for Vision‐Based Control in Robotic Heart Surgery

Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments

Twin-S: A Digital Twin for Skull-base Surgery

Cognitive tracking of surgical instruments based on stereo vision and depth sensing

Visualization and real-time tracking of bone drill by electromagnetic position sensors

Video-based Surgical Skills Assessment using Long term Tool Tracking

Soft-Tissue Motion Tracking And Structure Estimation For Robotic Assisted Mis Procedures

Vitreoretinal Surgical Instrument Tracking in Three Dimensions Using Deep Learning

ST-ITEF: Spatio-Temporal Intraoperative Task Estimating Framework to recognize surgical phase and predict instrument path based on multi-object tracking in keratoplasty

Gaze-Contingent Soft Tissue Deformation Tracking for Minimally Invasive Robotic Surgery

Vision-based Deformation Recovery for Intraoperative Force Estimation of Tool–tissue Interaction for Neurosurgery