Abstract:Purpose: The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception. Methods: We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception. Results: A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31 mm, a vertical image resolution of 0.82 mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3 mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87 mm) as the conventional system (1.95 ± 0.71 mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts. Conclusion: We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.

Augmented Reality Computational Spectral Imaging for Surgical Guidance

Augmented Reality Computational Spectral Imaging for Surgical Guidance

Augmented reality in the operating room: a clinical feasibility study

High-Quality See-Through Surgical Guidance System Using Enhanced 3-D Autostereoscopic Augmented Reality

Visualization, Registration and Tracking Techniques for Augmented Reality Guided Surgery: a Review

A Comparative Evaluation of Optical See-Through Augmented Reality in Surgical Guidance.

Seamless Augmented Reality Integration in Arthroscopy: A Pipeline for Articular Reconstruction and Guidance

A High-Accuracy Surgical Augmented Reality System Using Enhanced Integral Videography Image Overlay.

A Convenient Method of Video See-Through Augmented Reality Based on Image-Guided Surgery System

The Application of Augmented Reality Technology in Perioperative Visual Guidance: Technological Advances and Innovation Challenges

Augmented reality-based autostereoscopic surgical visualization system for telesurgery

Research on Stereo 3D Display for Augmented Reality Surgical Navigation

A Projector-Based Augmented Reality Navigation System for Computer-Assisted Surgery

Augmented-Reality Surgical Navigation System for Better Healthcare Visualization

Real-Time Computer-Generated Integral Imaging and 3d Image Calibration for Augmented Reality Surgical Navigation

3D augmented reality based orthopaedic interventions

Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer

Co-axial Projective Imaging for Augmented Reality Telementoring in Skin Cancer Surgery

Real-time Geometry-Aware Augmented Reality in Minimally Invasive Surgery

High-precision Three-dimensional Space See-through Surgical Navigation

An augmented reality-guided biopsy system using a high-speed motion tracking and real-time registration platform