Abstract:ObjectiveTo automate surgeon skills evaluation using robotic instrument kinematic data. Additionally, to implement an unsupervised mislabeling detection algorithm to identify potentially mislabeled samples that can be removed to improve model performance.MethodsVideo recordings and instrument kinematic data were derived from suturing exercises completed on the Mimic FlexVR robotic simulator. A structured human consensus-building process was developed to determine Robotic Anastomosis Competency Evaluation technical scores across 3 human graders. A 2-layer long short-term memory–based classification model used instrument kinematic data to automate suturing skills assessment. An unsupervised label analyzer (NoiseRank) was used to identify potential mislabeling of skills data. Performance of the long short-term memory model's technical skill score prediction was measured by best area under the curve over the training runs. NoiseRank outputted a ranked list of rated skills assessments based on likelihood of mislabeling.Results22 surgeons performed 226 suturing attempts, which were broken down into 1,404 individual skill assessment points. Automation of needle entry angle, needle driving, and needle withdrawal technical skill scores performed better (area under the curve 0.698–0.705) than needle positioning (0.532) at baseline using all available data. Potential mislabels were subsequently identified by NoiseRank and removed, improving model performance across all domains (area under the curve 0.551–0.766).ConclusionUsing ground truth labels from human graders and robotic instrument kinematic data, machine learning models have automated assessment of detailed suturing technical skills with good performance. Further, an unsupervised mislabeling detection algorithm projected mislabeled data, allowing for their removal and subsequent improvement of model performance.

Toward Personalized Training and Skill Assessment in Robotic Minimally Invasive Surgery

An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery

Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

Video-based Surgical Skills Assessment using Long term Tool Tracking

Towards Automated Surgical Skill Evaluation of Endovascular Catheterization Tasks Based on Force and Motion Signatures.

Towards Interpretable Motion-level Skill Assessment in Robotic Surgery

SATR-DL: Improving Surgical Skill Assessment and Task Recognition in Robot-assisted Surgery with Deep Neural Networks

Assessing system operation skills in robotic surgery trainees

Learning surgical skills under the RCM constraint from demonstrations in robot-assisted minimally invasive surgery

Automated Objective Surgical Skill Assessment in the Operating Room Using Unstructured Tool Motion

Uncertainty-Aware Self-Supervised Learning for Cross-Domain Technical Skill Assessment in Robot-Assisted Surgery

Using AI and computer vision to analyze technical proficiency in robotic surgery

Towards Accurate and Interpretable Surgical Skill Assessment: A Video-Based Method Incorporating Recognized Surgical Gestures and Skill Levels

Towards Accurate and Interpretable Surgical Skill Assessment: a Video-Based Method for Skill Score Prediction and Guiding Feedback Generation

Automated Surgical Skill Assessment in Endoscopic Pituitary Surgery using Real-time Instrument Tracking on a High-fidelity Bench-top Phantom

Road to automating robotic suturing skills assessment: Battling mislabeling of the ground truth

A touch-free human-robot collaborative surgical navigation robotic system based on hand gesture recognition

Mapping Surgeon's Hand/Finger Motion During Conventional Microsurgery to Enhance Intuitive Surgical Robot Teleoperation

Using hand pose estimation to automate open surgery training feedback

Objective Assessment of Robotic Surgical Technical Skill: A Systematic Review

Preliminary Design of a Robotic System for Kinaesthetic Training of Surgical Tasks