Abstract:Introduction: Remote military operations require rapid response times for effective relief and critical care. Yet, the military theater is under austere conditions, so communication links are unreliable and subject to physical and virtual attacks and degradation at unpredictable times. Immediate medical care at these austere locations requires semi-autonomous teleoperated systems, which enable the completion of medical procedures even under interrupted networks while isolating the medics from the dangers of the battlefield. However, to achieve autonomy for complex surgical and critical care procedures, robots require extensive programming or massive libraries of surgical skill demonstrations to learn effective policies using machine learning algorithms. Although such datasets are achievable for simple tasks, providing a large number of demonstrations for surgical maneuvers is not practical. This article presents a method for learning from demonstration, combining knowledge from demonstrations to eliminate reward shaping in reinforcement learning (RL). In addition to reducing the data required for training, the self-supervised nature of RL, in conjunction with expert knowledge-driven rewards, produces more generalizable policies tolerant to dynamic environment changes. A multimodal representation for interaction enables learning complex contact-rich surgical maneuvers. The effectiveness of the approach is shown using the cricothyroidotomy task, as it is a standard procedure seen in critical care to open the airway. In addition, we also provide a method for segmenting the teleoperator's demonstration into subtasks and classifying the subtasks using sequence modeling. Materials and methods: A database of demonstrations for the cricothyroidotomy task was collected, comprising six fundamental maneuvers referred to as surgemes. The dataset was collected by teleoperating a collaborative robotic platform-SuperBaxter, with modified surgical grippers. Then, two learning models are developed for processing the dataset-one for automatic segmentation of the task demonstrations into a sequence of surgemes and the second for classifying each segment into labeled surgemes. Finally, a multimodal off-policy RL with rewards learned from demonstrations was developed to learn the surgeme execution from these demonstrations. Results: The task segmentation model has an accuracy of 98.2%. The surgeme classification model using the proposed interaction features achieved a classification accuracy of 96.25% averaged across all surgemes compared to 87.08% without these features and 85.4% using a support vector machine classifier. Finally, the robot execution achieved a task success rate of 93.5% compared to baselines of behavioral cloning (78.3%) and a twin-delayed deep deterministic policy gradient with shaped rewards (82.6%). Conclusions: Results indicate that the proposed interaction features for the segmentation and classification of surgical tasks improve classification accuracy. The proposed method for learning surgemes from demonstrations exceeds popular methods for skill learning. The effectiveness of the proposed approach demonstrates the potential for future remote telemedicine on battlefields.

Mixed-Reality-Guided Teleoperation of a Collaborative Robot for Surgical Procedures

Mixed reality based teleoperation and visualization of surgical robotics

Collaborative Framework for Robot-Assisted Minimally Invasive Surgery Using a 7-Dof Anthropomorphic Robot

A Mixed-Reality Tele-Operation Method for High-Level Control of a Legged-Manipulator Robot

A Cooperative Control Framework for Haptic Guidance of Bimanual Surgical Tasks Based on Learning from Demonstration.

Augmented Reality Enabled Telepresence Framework for Intuitive Robotic Teleoperation

Towards Human-Robot Collaborative Surgery: Trajectory and Strategy Learning in Bimanual Peg Transfer

Integration of Virtual Reality in the Control System of an Innovative Medical Robot for Single-Incision Laparoscopic Surgery

Human-Robot Shared Control for Surgical Robot Based on Context-Aware Sim-to-Real Adaptation

Robotic Manipulator‐Assisted Omnidirectional Augmented Reality for Endoluminal Intervention Telepresence

Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interaction

ARei: Augmented-Reality-Assisted Touchless Teleoperated Robot for Endoluminal Intervention

Mixed reality human teleoperation with device-agnostic remote ultrasound: Communication and user interaction

Robot-Assisted Teleoperation Ultrasound System Based on Fusion of Augmented Reality and Predictive Force

Internet of Things (Iot)-Based Collaborative Control of a Redundant Manipulator for Teleoperated Minimally Invasive Surgeries

An augmented-reality-assisted immersive system for robotic arms teleoperation

Telemanipulation of Snake-Like Robots for Minimally Invasive Surgery of the Upper Airway

Shared Control for Bimanual Telesurgery with Optimized Robotic Partner

A Transparent Teleoperated Robotic Surgical System with Predictive Haptic Feedback and Force Modelling

ASAP-CORPS: A Semi-Autonomous Platform for COntact-Rich Precision Surgery

Teleoperated Surgical Robot with Adaptive Interactive Control Architecture for Tissue Identification