Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interaction

Yonghao Long,Chengkun Li,Qi Dou
2024-02-29
Abstract:With the growing popularity of robotic surgery, education becomes increasingly important and urgently needed for the sake of patient safety. However, experienced surgeons have limited accessibility due to their busy clinical schedule or working in a distant city, thus can hardly provide sufficient education resources for novices. Remote mentoring, as an effective way, can help solve this problem, but traditional methods are limited to plain text, audio, or 2D video, which are not intuitive nor vivid. Augmented reality (AR), a thriving technique being widely used for various education scenarios, is promising to offer new possibilities of visual experience and interactive teaching. In this paper, we propose a novel AR-based robotic surgery remote mentoring system with efficient 3D scene visualization and natural 3D hand interaction. Using a head-mounted display (i.e., HoloLens), the mentor can remotely monitor the procedure streamed from the trainee's operation side. The mentor can also provide feedback directly with hand gestures, which is in-turn transmitted to the trainee and viewed in surgical console as guidance. We comprehensively validate the system on both real surgery stereo videos and ex-vivo scenarios of common robotic training tasks (i.e., peg-transfer and suturing). Promising results are demonstrated regarding the fidelity of streamed scene visualization, the accuracy of feedback with hand interaction, and the low-latency of each component in the entire remote mentoring system. This work showcases the feasibility of leveraging AR technology for reliable, flexible and low-cost solutions to robotic surgical education, and holds great potential for clinical applications.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The paper primarily addresses the issues present in robotic surgery education and proposes a new solution. With the development of robotic surgery technology, it has become particularly important to effectively train novice doctors. However, experienced surgeons often find it challenging to provide sufficient guidance resources to novices due to busy work schedules or geographical constraints. Traditional remote guidance methods (such as text, audio, or 2D video) have limitations in terms of intuitiveness and interactivity, failing to fully meet the teaching needs of complex surgical scenarios. To solve the above problems, the authors propose a robotic surgery remote guidance system based on Augmented Reality (AR) technology. This system features efficient 3D scene visualization and natural 3D hand interaction capabilities. By using head-mounted displays (such as HoloLens), the instructor can remotely monitor the surgical process from the operation site and provide feedback directly through gestures. These gestures are transmitted in real-time to the trainee and displayed as guidance information on the surgical console. Experimental results show that the system performs excellently in terms of visualization fidelity of surgical scenes, accuracy of hand interactions, and low latency of the entire remote guidance system. In summary, this study aims to develop a reliable, flexible, and low-cost robotic surgery education solution using augmented reality technology. This will help improve the effectiveness of remote guidance and has the potential to be widely applied in clinical practice.