Abstract:To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future.

Capturing relationships between suturing sub-skills to improve automatic suturing assessment

Towards Unified Surgical Skill Assessment

AIxSuture: vision-based assessment of open suturing skills

Suturing Tasks Automation Based on Skills Learned From Demonstrations: A Simulation Study

Objective and Automated Quantification of Instrument Handling for Open Surgical Suturing Skill Assessment: A Simulation-Based Study

Measuring hand movement for suturing skill assessment: A simulation-based study

Deep learning prediction of error and skill in robotic prostatectomy suturing

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

Surgical Skill Assessment on In-Vivo Clinical Data Via the Clearness of Operating Field

Open-source High-precision Autonomous Suturing Framework With Visual Guidance

Road to automating robotic suturing skills assessment: Battling mislabeling of the ground truth

An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery

Video-based fully automatic assessment of open surgery suturing skills

Clearness of Operating Field: a Surrogate for Surgical Skills on in Vivo Clinical Data

Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria

Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

Towards Augmented Reality-based Suturing in Monocular Laparoscopic Training

The current state of autonomous suturing: a systematic review

Toward Image-Guided Automated Suture Grasping Under Complex Environments: A Learning-Enabled and Optimization-Based Holistic Framework

Autonomous Image-to-Grasp Robotic Suturing Using Reliability-Driven Suture Thread Reconstruction

Assessing system operation skills in robotic surgery trainees