Abstract:The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

Robust Vision Using Retro Reflective Markers for Remote Handling in ITER

Design and manufacturing of an optimized retro reflective marker for photogrammetric pose estimation in ITER

Monocular Robust Depth Estimation Vision System for Robotic Tasks Interventions in Metallic Targets

Autonomous Recognition and Positioning of Shield Segments Based on Red, Green, Blue and Depth Information

Inside-out Infrared Marker Tracking via Head Mounted Displays for Smart Robot Programming

Markerless Inside-Out Tracking for Interventional Applications

Design and implementation of eye tracking application for generation of augmented reality content on spatial see through display of remote and virtual control tower (RVT)

A Novel Marker Tracking Method Based on Extended Kalman Filter for Multi-Camera Optical Tracking Systems

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Robot-assisted system for non-invasive wide-range flexible eye positioning and tracking in particle radiotherapy

Robust Object Pose Tracking for Augmented Reality Guidance and Teleoperation

Designing for Remote Handling: the case-study of the ITER Plasma Position Reflectometry in-vessel antennas

AprilTags 3D: Dynamic Fiducial Markers for Robust Pose Estimation in Highly Reflective Environments and Indirect Communication in Swarm Robotics

Autonomous 3D geometry reconstruction through robot-manipulated optical sensors

Application of a Vision-Based Single Target on Robot Positioning System

Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives

Dynamic Virtual Fixture Generation Based on Intra-Operative 3D Image Feedback in Robot-Assisted Minimally Invasive Thoracic Surgery

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

Computer Vision-Aided Reconfigurable Intelligent Surface-Based Beam Tracking: Prototyping and Experimental Results

Wearable Surgical Optical Tracking System Based on Multi-Modular Sensor Fusion

MarkerPose: Robust Real-time Planar Target Tracking for Accurate Stereo Pose Estimation