Abstract:Purpose: The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception. Methods: We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception. Results: A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31 mm, a vertical image resolution of 0.82 mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3 mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87 mm) as the conventional system (1.95 ± 0.71 mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts. Conclusion: We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.

3D Medical Electronic Endoscope System Based on Binocular Stereo Vision

A Three-Dimensional Measurement Method for Binocular Endoscopes Based on Deep Learning

Three-Dimensional Stitching of Binocular Endoscopic Images Based on Feature Points

Human-Computer Interaction System Based on Binocular Stereo Vision

Floating autostereoscopic 3D display with multidimensional images for telesurgical visualization

Spatial 3-D Display System Based On Sequential Effect

Stereovision System For 3-D Measurement Of Mems Components

3D Autostereoscopic Medical Image Display with Long Visualization Depth

High-Quality See-Through Surgical Guidance System Using Enhanced 3-D Autostereoscopic Augmented Reality

Bifocal computational near eye light field displays and Structure parameters determination scheme for bifocal computational display.

Image generation system and method for stereoscopic 3D display

High Quality Autostereoscopic Surgical Display Using Anti-aliased Integral Videography Imaging

3 D interactive surgical visualization system 2 3 D Interactive Surgical Visualization System using Mobile Spatial Information Acquisition and Autostereoscopic Display

Augmented reality-based autostereoscopic surgical visualization system for telesurgery

A Time-of-Flight Stereoscopic Endoscope for Anatomical 3D Reconstruction

3D Interactive Surgical Visualization System Using Mobile Spatial Information Acquisition and Autostereoscopic Display

High quality 3D autostereoscopic medical display method with optical enhancement and image compensation

A Monocular Variable Magnifications 3D Laparoscope System Using Double Liquid Lenses

A 3-D mixed-reality system for stereoscopic visualization of medical dataset

Stereoscopic head-mounted display incorporated into microsurgical procedures: technical note

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection