Abstract:Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the redefinition of different human-machine interfaces. We demonstrate that this AR paradigm is generic, and can benefit a wide variety of procedures. Our system achieved an error of $4.76\pm2.91$ mm for placing K-wire in a fracture management procedure, and yielded errors of $1.57\pm1.16^\circ$ and $1.46\pm1.00^\circ$ in the abduction and anteversion angles, respectively, for total hip arthroplasty. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes.

HCI Based on Gesture Recognition in an Augmented Reality System for Diagnosis Planning and Training.

A Bare Hand Controlled Ar Map Navigation System

An Interactive Task Analysis Framework and Interactive System Research for Computer Aided Diagnosis

Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

3D Intuitive Gesture Interaction via Motion Sensing

An interaction system using mixed hand gestures.

Video camera-based dynamic gesture recognition for HCI

Gesture Interaction System Design for Telerehabilitation Based on Mediapipe

Design and evaluation of a hand gesture recognition approach for real-time interactions

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

Design of hand gesture recognition system for human-computer interaction

Gesture-based target acquisition in virtual and augmented reality.

Vision-Based Real-Time Human–Computer Interaction on Hand Gesture Recognition

HCI on the Table: Robust Gesture Recognition Using Acoustic Sensing in Your Hand

Current Status and Challenge for Implementation in Clinical Application of Augmented Reality in Orthopedics Department at a Medical Center

Hand gesture based control with multi-modality data - towards surgical applications

Interaction With Gaze, Gesture, and Speech in a Flexibly Configurable Augmented Reality System

Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interaction

Game-based human computer interaction using gesture recognition for rehabilitation

Automated Patient Positioning with Learned 3D Hand Gestures

Tangible and Mid-Air Interactions in Hand-Held Augmented Reality for Upper Limb Rehabilitation: An Evaluation of User Experience and Motor Performance