Abstract:Purpose This paper aims to investigate the problem of vision based autonomous laparoscope control, which can serve as the primary function for semi-autonomous minimally invasive surgical robot system. Providing the surgical gesture recognition information is a fundamental key component for enabling intelligent context-aware assistance in autonomous laparoscope control task. While significant advances have been made in recent years, how to effectively carry out the efficient integration of surgical gesture recognition and autonomous laparoscope control algorithms for robotic assisted minimally invasive surgical robot system is still an open and challenging topic. Design/methodology/approach The authors demonstrate a novel surgeon in-loop semi-autonomous robotic-assisted minimally invasive surgery framework by integrating the surgical gesture recognition and autonomous laparoscope control tasks. Specifically, they explore using a transformer-based deep convolutional neural network to effectively recognize the current surgical gesture. Next, they propose an autonomous laparoscope control model to provide optimal field of view which is in line with surgeon intra-operation preferences. Findings The effectiveness of this surgical gesture recognition methodology is demonstrated on the public JIGSAWS and Cholec80 data sets, outperforming the comparable state-of-the-art methods. Furthermore, the authors have validated the effectiveness of the proposed semi-autonomous framework on the developed HUAQUE surgical robot platforms. Originality/value This study demonstrates the feasibility to perform cognitive assistant human–robot shared control for semi-autonomous robotic-assisted minimally invasive surgery, contributing to the reference for further surgical intelligence in computer-assisted intervention systems.

Gesture Control Method for Oral Blood Mixed Liquid Removal Robot*

Autonomous Aspirating Robot for Removing Saliva Blood Mixed Liquid in Oral Surgery

Image-Guided Teleoperation for Soft Bronchoscopy Robot

Development and Control of A Robotic Arm for Percutaneous Surgery

A touch-free human-robot collaborative surgical navigation robotic system based on hand gesture recognition

Autonomous Robot for Removing Superficial Traumatic Blood

Hands-free Robot Control System Using Mouth Tracking

Appearance-Based Gaze Estimator for Natural Interaction Control of Surgical Robots

GuLiM: A Hybrid Motion Mapping Technique for Teleoperation of Medical Assistive Robot in Combating the COVID-19 Pandemic

Design and Voice-Based Control of a Nasal Endoscopic Surgical Robot

Design of Restaurant Intelligent Beverage Service Robot Based on Gesture Control

Hand gesture based control with multi-modality data - towards surgical applications

A collaborative robot for COVID-19 oropharyngeal swabbing

Gestonurse: a robotic surgical nurse for handling surgical instruments in the operating room

Autonomous Robotic Suction to Clear the Surgical Field for Hemostasis using Image-based Blood Flow Detection

Machine Learning-Based Surgical State Perception and Collaborative Control for a Vascular Interventional Robot

Design and Implementation of a Robotic System for Oropharyngeal Swab Sampling Application

Reinforcement Learning Based Manipulation Skill Transferring for Robot-assisted Minimally Invasive Surgery.

SGR-AutoLap: Surgical Gesture Recognition-Based Autonomous Laparoscope Control for Human-Robot Shared Control in Semi-Autonomous Minimally Invasive Surgical Robot

Multi-Sensor Guided Hand Gesture Recognition for a Teleoperated Robot Using a Recurrent Neural Network

An Eye-Gaze-Controlled Needle Deployment Robot: Design, Modeling, and Experimental Evaluation