Abstract:Background: In recent years, computer-assisted intervention and robot-assisted surgery are receiving increasing attention. The need for real-time identification and tracking of surgical tools and tool tips is constantly demanding. A series of researches focusing on surgical tool tracking and identification have been performed. However, the size of dataset, the sensitivity/precision, and the response time of these studies were limited. In this work, we developed and utilized an automated method based on Convolutional Neural Network (CNN) and You Only Look Once (YOLO) v3 algorithm to locate and identify surgical tools and tool tips covering five different surgical scenarios. Materials and methods: An algorithm of object detection was applied to identify and locate the surgical tools and tool tips. DarkNet-19 was used as Backbone Network and YOLOv3 was modified and applied for the detection. We included a series of 181 endoscopy videos covering 5 different surgical scenarios: pancreatic surgery, thyroid surgery, colon surgery, gastric surgery, and external scenes. A total amount of 25,333 images containing 94,463 targets were collected. Training and test sets were divided in a proportion of 2.5:1. The data sets were openly stored at the Kaggle database. Results: Under an Intersection over Union threshold of 0.5, the overall sensitivity and precision rate of the model were 93.02% and 89.61% for tool recognition and 87.05% and 83.57% for tool tip recognition, respectively. The model demonstrated the highest tool and tool tip recognition sensitivity and precision rate under external scenes. Among the four different internal surgical scenes, the network had better performances in pancreatic and colon surgeries and poorer performances in gastric and thyroid surgeries. Conclusion: We developed a surgical tool and tool tip recognition model based on CNN and YOLOv3. Validation of our model demonstrated satisfactory precision, accuracy, and robustness across different surgical scenes.

Real Time Surgical Instrument Object Detection Using YOLOv7

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Surgical Instrument Detection Algorithm Based on Improved YOLOv7x

Surgical Instrument Detection for Laparoscopic Surgery using Deep Learning

Situation Awareness for Automated Surgical Check-listing in AI-Assisted Operating Room

SINet: A hybrid deep CNN model for real-time detection and segmentation of surgical instruments

A Deep Learning Approach for Surgical Instruments Detection System in Total Knee Arthroplasty - Automatic Creation of Training Data and Reduction of Training Time -

YOLOv7-Based Multiple Surgical Tool Localization and Detection in Laparoscopic Videos

Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network

Real-Time Surgical Tool Detection in Minimally Invasive Surgery Based on Attention-Guided Convolutional Neural Network

Real-time Surgical Instrument Detection in Robot-Assisted Surgery Using a Convolutional Neural Network Cascade

Deep Residual Learning for Instrument Segmentation in Robotic Surgery

Automatic Surgical Items Tracking in Laparoscopic Surgery Utilizing Deep Learning for Retained Items Detection and Classification

Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method.

Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery

Towards Surgical Tools Detection and Operative Skill Assessment Based on Deep Learning

Robust Surgical Tools Detection in Endoscopic Videos with Noisy Data

Deep Learning for Surgical Instrument Recognition and Segmentation in Robotic-Assisted Surgeries: A Systematic Review

Application and evaluation of surgical tool and tool tip recognition based on Convolutional Neural Network in multiple endoscopic surgical scenarios

Automatic Surgical Instrument Recognition—A Case of Comparison Study between the Faster R-CNN, Mask R-CNN, and Single-Shot Multi-Box Detectors

Weakly Supervised YOLO Network for Surgical Instrument Localization in Endoscopic Videos