Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

Ziheng Wang,Ann Majewicz Fey

DOI: https://doi.org/10.1007/s11548-018-1860-1

2019-03-07

Abstract:With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels. We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved a competitive accuracy of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully-tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within 1-3 second window, without needing an observation of entire training trial. This study highlights the potentials of deep architectures for an proficient online skill assessment in modern surgical training.

Computer Vision and Pattern Recognition,Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve objective skill assessment in robot - assisted surgery. Specifically, existing skill assessment methods usually need to convert the kinematic data of robots into intermediate features or gesture segments. This method is not only costly and inefficient, but also requires a great deal of domain - specific knowledge. To solve these problems, the author proposes an analysis framework based on deep learning, especially using convolutional neural networks (CNN), to directly map kinematic features from multivariate time - series data to individual skill levels, thereby achieving end - to - end learning without manually designing features. This method aims to improve the accuracy, efficiency and reliability of assessment. Especially in minimally invasive surgical training, it can reliably interpret skills within a 1 - 3 - second time window without the need to observe the entire training trial process. In addition, this study also explores the application of data augmentation techniques to overcome the over - fitting problem caused by small - scale data sets and improve the generalization ability of the model. Verified by experiments on the publicly available minimally invasive surgical robot data set JIGSAWS, the accuracies of this model in the three standard training tasks of suturing, needle - passing and knot - tying reached 92.5%, 95.4% and 91.3% respectively, demonstrating its potential in modern surgical training.

Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

SATR-DL: Improving Surgical Skill Assessment and Task Recognition in Robot-assisted Surgery with Deep Neural Networks

An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery

Video-based surgical skill assessment using 3D convolutional neural networks

Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria

Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks

Evaluating surgical skills from kinematic data using convolutional neural networks

Deep Learning to Automate Technical Skills Assessment in Robotic Surgery

Towards Accurate and Interpretable Surgical Skill Assessment: A Video-Based Method Incorporating Recognized Surgical Gestures and Skill Levels

Deep learning prediction of error and skill in robotic prostatectomy suturing

Towards Accurate and Interpretable Surgical Skill Assessment: a Video-Based Method for Skill Score Prediction and Guiding Feedback Generation

Quantification of Robotic Surgeries with Vision-Based Deep Learning

Toward Personalized Training and Skill Assessment in Robotic Minimally Invasive Surgery

Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data

The development of an eye movement-based deep learning system for laparoscopic surgical skills assessment

Towards Unified Surgical Skill Assessment

CWT-ViT: A Time-Frequency Representation and Vision Transformer-Based Framework for Automated Robotic Surgical Skill Assessment

Surgical Skill Assessment on In-Vivo Clinical Data Via the Clearness of Operating Field

SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction

Uncertainty-Aware Self-Supervised Learning for Cross-Domain Technical Skill Assessment in Robot-Assisted Surgery

Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery