Abstract:The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

Human–machine integration based augmented reality assisted wire-bending training system for orthodontics

A general framework of augmented reality aided teleoperation guidance

Gesture Recognition in Augmented Reality Assisted Assembly Training

Development of a robotic system for orthodontic archwire bending

Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interaction

Enhancing Visualisation of Anatomical Presentation and Education Using Marker-based Augmented Reality Technology on Web-based Platform

A Wearable Augmented Reality Navigation System for Surgical Telementoring Based on Microsoft HoloLens

Y-AR: A Mixed Reality CAD Tool for 3D Wire Bending

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

A Multi-Sensor Interface to Improve the Learning Experience in Arc Welding Training Tasks

Augmented Reality-based Feedback for Technician-in-the-loop C-arm Repositioning

A Visual Contour-based Detection Method for Orthodontic Archwire During Robotic Bending

Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

A Distributed Augmented Reality System for Medical Training and Simulation

Current Status and Challenge for Implementation in Clinical Application of Augmented Reality in Orthopedics Department at a Medical Center

An Human-Computer Interactive Augmented Reality System for Coronary Artery Diagnosis Planning and Training

Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario

3D augmented reality based orthopaedic interventions

Object Detection and Text Recognition for Immersive Augmented Reality Training in Laser Powder Bed Fusion

Multi-Channel Augmented Reality Interactive Framework Design for Ship Outfitting Guidance