Abstract:Background: We developed a 3D camera system to track motion in a surgical field. This system has the potential to introduce augmented reality (AR) systems non-invasively, eliminating the need for the invasive AR markers conventionally required. The present study was performed to verify the real-time tracking accuracy of this system, assess the feasibility of integrating this system into the surgical workflow, and establish its potential to enhance the accuracy and efficiency of orthopedic procedures. Methods: To evaluate the accuracy of AR technology using a 3D camera, a forearm bone model was created. The forearm model was depicted using a 3D camera, and its accuracy was verified in terms of the positional relationship with a 3D bone model created from previously imaged CT data. Images of the surgical field (capturing the actual forearm) were taken and saved in nine poses by rotating the forearm from pronation to supination. The alignment of the reference points was computed at the three points of CT versus the three points of the 3D camera, yielding a 3D rotation matrix representing the positional relationship. In the original system, a stereo vision-based 3D camera, with a depth image resolution of 1280×720 pixels, 30 frames per second, and a lens field of view of 64 specifications, with a baseline of 3 cm, capable of optimally acquiring real-time 3D data at a distance of 40-60 cm from the subject was used. In the modified system, the following modifications were made to improve tracking performance: (1) color filter processing was changed from HSV to RGB, (2) positional detection accuracy was modified with supporting marker sizes of 8 mm in diameter, and (3) the detection of marker positions was stabilized by calculating the marker position for each frame. Tracking accuracy was examined with the original system and modified system for the following parameters: differences in the rotation matrix, maximum and minimum inter-reference point errors between CT-based and camera-based 3D data, and the average error for the three reference points. Results: In the original system, the average difference in rotation matrices was 5.51±2.68 mm. Average minimum and maximum errors were 1.10±0.61 and 15.53±12.51 mm, respectively. The average error of reference points was 6.26±4.49 mm. In the modified system, the average difference in rotation matrices was 4.22±1.73 mm. Average minimum and maximum errors were 0.79±0.49 and 1.94±0.87 mm, respectively. The average error of reference points was 1.41±0.58 mm. In the original system, once tracking failed, it was difficult to recover tracking accuracy. This resulted in a large maximum error in supination positions. These issues were resolved by the modified system. Significant improvements were achieved in maximum errors and average errors using the modified system (P<0.05). Conclusion: AR technology using a 3D camera was developed. This system allows direct comparisons of 3D data from preoperative CT scans with 3D data acquired from the surgical field using a 3D camera. This method has the advantage of introducing AR into the surgical field without invasive markers.

Augmented Reality to Assist in the Diagnosis of Temporomandibular Joint Alterations

Augmented reality in the operating room: a clinical feasibility study

Early Feasibility Studies of Augmented Reality Navigation for Lateral Skull Base Surgery

Enhancing Visualisation of Anatomical Presentation and Education Using Marker-based Augmented Reality Technology on Web-based Platform

Vision-based Markerless Registration Using Stereo Vision and an Augmented Reality Surgical Navigation System: a Pilot Study.

Mixed Reality and Artificial Intelligence: a Holistic Approach to Multimodal Visualization and Extended Interaction in Knee Osteotomy

A practical marker-less image registration method for augmented reality oral and maxillofacial surgery

Augmented Reality Registration System for Visualization of Skull Landmarks

Development of Augmented Reality Vision for Osteosynthesis Using a 3D Camera

Novel Use of Virtual Reality and Augmented Reality in Temporomandibular Total Joint Replacement Using Stock Prosthesis

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

Application of image recognition-based tracker-less augmented reality navigation system in a series of sawbone trials

Evaluating the feasibility of using augmented reality for tooth preparation

Real-time in Situ Three-Dimensional Integral Videography and Surgical Navigation Using Augmented Reality: a Pilot Study

Augmented reality in implantology: virtual surgical checklist and augmented implant placement

Closing the Calibration Loop: An Inside-out-tracking Paradigm for Augmented Reality in Orthopedic Surgery

The Preliminary Application of Augmented Reality in Unilateral Orbitozygomatic Maxillary Complex Fractures Treatment

Is Total Knee Arthroplasty Surgical Performance Enhanced Using Augmented Reality? A Single-Center Study on 76 Consecutive Patients

Augmented reality calibration using feature triangulation iteration-based registration for surgical navigation

Mandibular resection and defect reconstruction guided by a contour registration-based augmented reality system: A preclinical trial