Abstract:Background: In recent years, computer-assisted intervention and robot-assisted surgery are receiving increasing attention. The need for real-time identification and tracking of surgical tools and tool tips is constantly demanding. A series of researches focusing on surgical tool tracking and identification have been performed. However, the size of dataset, the sensitivity/precision, and the response time of these studies were limited. In this work, we developed and utilized an automated method based on Convolutional Neural Network (CNN) and You Only Look Once (YOLO) v3 algorithm to locate and identify surgical tools and tool tips covering five different surgical scenarios. Materials and methods: An algorithm of object detection was applied to identify and locate the surgical tools and tool tips. DarkNet-19 was used as Backbone Network and YOLOv3 was modified and applied for the detection. We included a series of 181 endoscopy videos covering 5 different surgical scenarios: pancreatic surgery, thyroid surgery, colon surgery, gastric surgery, and external scenes. A total amount of 25,333 images containing 94,463 targets were collected. Training and test sets were divided in a proportion of 2.5:1. The data sets were openly stored at the Kaggle database. Results: Under an Intersection over Union threshold of 0.5, the overall sensitivity and precision rate of the model were 93.02% and 89.61% for tool recognition and 87.05% and 83.57% for tool tip recognition, respectively. The model demonstrated the highest tool and tool tip recognition sensitivity and precision rate under external scenes. Among the four different internal surgical scenes, the network had better performances in pancreatic and colon surgeries and poorer performances in gastric and thyroid surgeries. Conclusion: We developed a surgical tool and tool tip recognition model based on CNN and YOLOv3. Validation of our model demonstrated satisfactory precision, accuracy, and robustness across different surgical scenes.

YOLOv7-Based Multiple Surgical Tool Localization and Detection in Laparoscopic Videos

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Application and evaluation of surgical tool and tool tip recognition based on Convolutional Neural Network in multiple endoscopic surgical scenarios

Situation Awareness for Automated Surgical Check-listing in AI-Assisted Operating Room

Vision-Based Neurosurgical Guidance: Unsupervised Localization and Camera-Pose Prediction

Automated Detection of Small Bowel Lesions Based on Capsule Endoscopy Using Deep Learning Algorithm

Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks

SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos

Robust Surgical Tools Detection in Endoscopic Videos with Noisy Data

Surgical Tools Detection Based on Modulated Anchoring Network in Laparoscopic Videos

Weakly Supervised YOLO Network for Surgical Instrument Localization in Endoscopic Videos

Surgical Instrument Detection Algorithm Based on Improved YOLOv7x

Open surgery tool classification and hand utilization using a multi-camera system

CholecTrack20: A Dataset for Multi-Class Multiple Tool Tracking in Laparoscopic Surgery

Real-Time Tool Localization for Laparoscopic Surgery Using Convolutional Neural Network

Surgical Tools Detection Based on Training Sample Adaptation in Laparoscopic Videos

Deep learning visual analysis in laparoscopic surgery: a systematic review and diagnostic test accuracy meta-analysis

Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection

Automatic Surgical Items Tracking in Laparoscopic Surgery Utilizing Deep Learning for Retained Items Detection and Classification

Future-State Predicting LSTM for Early Surgery Type Recognition

Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network