Abstract:Purpose This paper aims to investigate the problem of vision based autonomous laparoscope control, which can serve as the primary function for semi-autonomous minimally invasive surgical robot system. Providing the surgical gesture recognition information is a fundamental key component for enabling intelligent context-aware assistance in autonomous laparoscope control task. While significant advances have been made in recent years, how to effectively carry out the efficient integration of surgical gesture recognition and autonomous laparoscope control algorithms for robotic assisted minimally invasive surgical robot system is still an open and challenging topic. Design/methodology/approach The authors demonstrate a novel surgeon in-loop semi-autonomous robotic-assisted minimally invasive surgery framework by integrating the surgical gesture recognition and autonomous laparoscope control tasks. Specifically, they explore using a transformer-based deep convolutional neural network to effectively recognize the current surgical gesture. Next, they propose an autonomous laparoscope control model to provide optimal field of view which is in line with surgeon intra-operation preferences. Findings The effectiveness of this surgical gesture recognition methodology is demonstrated on the public JIGSAWS and Cholec80 data sets, outperforming the comparable state-of-the-art methods. Furthermore, the authors have validated the effectiveness of the proposed semi-autonomous framework on the developed HUAQUE surgical robot platforms. Originality/value This study demonstrates the feasibility to perform cognitive assistant human–robot shared control for semi-autonomous robotic-assisted minimally invasive surgery, contributing to the reference for further surgical intelligence in computer-assisted intervention systems.

Automatic Adjustment of Laparoscopic Pose Using Deep Reinforcement Learning

Laparoscope Arm Automatic Positioning for Robot-Assisted Surgery Based on Reinforcement Learning

Development and validation of a deep-learning based assistance system for enhancing laparoscopic control level

Learning Laparoscope Actions via Video Features for Proactive Robotic Field-of-View Control

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Deep Reinforcement Learning Based Semi-Autonomous Control for Robotic Surgery

Deep Reinforcement Learning in Surgical Robotics: Enhancing the Automation Level

Open-Loop Motion Control of a Hydraulic Soft Robotic Arm Using Deep Reinforcement Learning

3D Perception based Imitation Learning under Limited Demonstration for Laparoscope Control in Robotic Surgery

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Data-driven Holistic Framework for Automated Laparoscope Optimal View Control with Learning-based Depth Perception

Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

Multi-degree-of-freedom Intelligent Ultrasound Robot System Based on Reinforcement Learning

SGR-AutoLap: Surgical Gesture Recognition-Based Autonomous Laparoscope Control for Human-Robot Shared Control in Semi-Autonomous Minimally Invasive Surgical Robot

Human-in-the-Loop Embodied Intelligence With Interactive Simulation Environment for Surgical Robot Learning

Deep reinforcement learning in continuous action space for autonomous robotic surgery

Deep-Learning-Based Force Sensing Method for a Flexible Endovascular Surgery Robot

Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control

A Physical Human-Robot Interaction Algorithm Based on Fuzzy Reinforcement Learning for Minimally Invasive Surgery Manipulator

Reinforcement Learning Based Manipulation Skill Transferring for Robot-assisted Minimally Invasive Surgery.

Automatic Guidance of Laparoscope Based on the Region of Interest for Robot Assisted Laparoscopic Surgery