Towards Surgical Tools Detection and Operative Skill Assessment Based on Deep Learning

Yuying Liu,Zijian Zhao,Pan Shi,Feng Li

DOI: https://doi.org/10.1109/tmrb.2022.3145672

2022-01-01

IEEE Transactions on Medical Robotics and Bionics

Abstract:Surgical tool detection and automatic operation skill assessment (AOSA) have important and extensive application scenarios in minimally invasive surgery (MIS). However, most of the deep learning methods currently used for surgical tool detection cannot achieve a good balance between speed and accuracy. We propose a new real-time detection algorithm for MIS tools, which called depth-wise separable convolutional network with convolutional long short-term memory (DSCNet-CLSTM). The network combines the advantages of the one-stage multi-scale feature maps’ concept in the state-of-art detection methods and the convolutional variant of the LSTM. This combination makes full use of the complementary information of spatial and temporal features learned from the laparoscopic video frames. In addition, we have established AOSA system. By processing the output information of the MIS tool detection algorithm, we can obtain the operation tool usage information in the laparoscopic video. Then, this information is taken as the dataset of AOSA system, which makes it possible to realize AOSA based on convolutional neural network (CNN). The proposed method achieved the mAP values of 100, 90.07 and 89.96% at a speed of 50.0 fps for the Endovis Challenge, ATLAS Dione, and Cholec80-locations datasets, respectively. The AOSA system obtained the mean squared error of 2.281, 0.987, 0.069, and 0.009 for the action timeline, heat map, motion trajectory, and all, respectively. The experimental results prove that the framework can be efficiently trained in an end-to-end manner and improves the algorithm’s detection accuracy and speed. Finally, we verify the feasibility of the designed AOSA system through experiments.