Abstract:To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future.

A Novel Method for Knot-Tying in Autonomous Robotic Assisted Surgery Using Deep Learning

Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

Toward Image-Guided Automated Suture Grasping Under Complex Environments: A Learning-Enabled and Optimization-Based Holistic Framework

Automated Surgical Knot Tying on Mini-Incision with Micro-Suture Based on Dual-Arm Nanorobot under Stereo Microscope

SATR-DL: Improving Surgical Skill Assessment and Task Recognition in Robot-assisted Surgery with Deep Neural Networks

Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery

Multi-stage Suture Detection for Robot Assisted Anastomosis based on Deep Learning

In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning

A Cascaded Neural Network System For Rating Student Performance In Surgical Knot Tying Simulation

Uncertainty-Aware Self-Supervised Learning for Cross-Domain Technical Skill Assessment in Robot-Assisted Surgery

A New Tensioning Method using Deep Reinforcement Learning for Surgical Pattern Cutting

Quantification of Robotic Surgeries with Vision-Based Deep Learning

Reinforcement Learning Based Manipulation Skill Transferring for Robot-assisted Minimally Invasive Surgery.

A Novel Model-Independent Approach for Autonomous Retraction of Soft Tissue

TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

Open-source High-precision Autonomous Suturing Framework With Visual Guidance

Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks

Suturing Tasks Automation Based on Skills Learned From Demonstrations: A Simulation Study

Manipulating Soft Tissues by Deep Reinforcement Learning for Autonomous Robotic Surgery

Collaborative Robot-Assisted Endovascular Catheterization with Generative Adversarial Imitation Learning

A Neural Network-based Suture-tension Estimation Method Using Spatio-temporal Features of Visual Information and Robot-state Information for Robot-assisted Surgery