Abstract:To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future.

TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

Untangling Dense Knots by Learning Task-Relevant Keypoints

In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning

A Novel Method for Knot-Tying in Autonomous Robotic Assisted Surgery Using Deep Learning

Learning Cloth Folding Tasks with Refined Flow Based Spatio-Temporal Graphs

Folding Knots Using a Team of Aerial Robots

Bimanual rope manipulation skill synthesis through context dependent correction policy learning from human demonstration

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Precise Robotic Needle-Threading with Tactile Perception and Reinforcement Learning

Automated Surgical Knot Tying on Mini-Incision with Micro-Suture Based on Dual-Arm Nanorobot under Stereo Microscope

Toward Image-Guided Automated Suture Grasping Under Complex Environments: A Learning-Enabled and Optimization-Based Holistic Framework

Watch and Act: Learning Robotic Manipulation from Visual Demonstration.

Efficient Robot Skill Learning with Imitation from a Single Video for Contact-Rich Fabric Manipulation

DeMoBot: Deformable Mobile Manipulation with Vision-based Sub-goal Retrieval

Learning Robot Manipulation from Cross-Morphology Demonstration

Learning to Manipulate Tools by Aligning Simulation to Video Demonstration

Visual Imitation Made Easy

Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

Lessons from Learning to Spin "Pens"