Abstract:Many keyhole interventions rely on bi-manual handling of surgical instruments, forcing the main surgeon to rely on a second surgeon to act as a camera assistant. In addition to the burden of excessively involving surgical staff, this may lead to reduced image stability, increased task completion time and sometimes errors due to the monotony of the task. Robotic endoscope holders, controlled by a set of basic instructions, have been proposed as an alternative, but their unnatural handling may increase the cognitive load of the (solo) surgeon, which hinders their clinical acceptance. More seamless integration in the surgical workflow would be achieved if robotic endoscope holders collaborated with the operating surgeon via semantically rich instructions that closely resemble instructions that would otherwise be issued to a human camera assistant, such as "focus on my right-hand instrument". As a proof of concept, this paper presents a novel system that paves the way towards a synergistic interaction between surgeons and robotic endoscope holders. The proposed platform allows the surgeon to perform a bimanual coordination and navigation task, while a robotic arm autonomously performs the endoscope positioning tasks. Within our system, we propose a novel tooltip localization method based on surgical tool segmentation and a novel visual servoing approach that ensures smooth and appropriate motion of the endoscope camera. We validate our vision pipeline and run a user study of this system. The clinical relevance of the study is ensured through the use of a laparoscopic exercise validated by the European Academy of Gynaecological Surgery which involves bi-manual coordination and navigation. Successful application of our proposed system provides a promising starting point towards broader clinical adoption of robotic endoscope holders.

Toward Human-Out-of-the-Loop Endoscope Navigation Based on Context Awareness for Enhanced Autonomy in Robotic Surgery

Robotic Endoscope Control via Autonomous Instrument Tracking

A Surgeon Preference-Guided Autonomous Instrument Tracking Method With a Robotic Flexible Endoscope Based on dVRK Platform

Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study

A lumen-adapted navigation scheme with spatial awareness from monocular vision for autonomous robotic endoscopy

Human-Robot Shared Control for Surgical Robot Based on Context-Aware Sim-to-Real Adaptation

Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand–Eye Coordination, and Image Definition

Data-driven Holistic Framework for Automated Laparoscope Optimal View Control with Learning-based Depth Perception

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery

Integrated Planning and Control of Robotic Surgical Instruments for Tasks Autonomy

Autonomous Intraluminal Navigation of a Soft Robot using Deep-Learning-based Visual Servoing

Visual servo control of endoscope-holding robot based on multi-objective optimization: System modeling and instrument tracking

Homography-based Visual Servoing with Remote Center of Motion for Semi-autonomous Robotic Endoscope Manipulation

Autonomous Laparoscope Control for Minimally Invasive Surgery With Intuition and RCM Constraints

Unconstrained lightweight control interface for robot-assisted minimally invasive surgery using MediaPipe framework and head-mounted display

Constrained Motion Planning for a Robotic Endoscope Holder based on Hierarchical Quadratic Programming

MEDiC: Autonomous Surgical Robotic Assistance to Maximizing Exposure for Dissection and Cautery

Orientation Matters: 6-DoF Autonomous Camera Movement for Minimally Invasive Surgery

Mixed reality based teleoperation and visualization of surgical robotics

The security of minimally invasive surgery with an autonomous flexible endoscope

Virtual Reality-based Control of Robotic Endoscope in Laparoscopic Surgery