Abstract:Objective: To investigate the feasibility and accuracy of computer vision-based artificial intelligence technology in detecting and recognizing instruments and organs in the scenario of radical laparoscopic gastrectomy for gastric cancer. Methods: Eight complete laparoscopic distal radical gastrectomy surgery videos were collected from four large tertiary hospitals in China (First Medical Center of Chinese PLA General Hospital [three cases], Liaoning Cancer Hospital [two cases], Liyang Branch of Jiangsu Province People's Hospital [two cases], and Fudan University Shanghai Cancer Center [one case]). PR software was used to extract frames every 5-10 seconds and convert them into image frames. To ensure quality, deduplication was performed manually to remove obvious duplication and blurred image frames. After conversion and deduplication, there were 3369 frame images with a resolution of 1,920×1,080 PPI. LabelMe was used for instance segmentation of the images into the following 23 categories: veins, arteries, sutures, needle holders, ultrasonic knives, suction devices, bleeding, colon, forceps, gallbladder, small gauze, Hem-o-lok, Hem-o-lok appliers, electrocautery hooks, small intestine, hepatogastric ligaments, liver, omentum, pancreas, spleen, surgical staplers, stomach, and trocars. The frame images were randomly allocated to training and validation sets in a 9:1 ratio. The YOLOv8 deep learning framework was used for model training and validation. Precision, recall, average precision (AP), and mean average precision (mAP) were used to evaluate detection and recognition accuracy. Results: The training set contained 3032 frame images comprising 30 895 instance segmentation counts across 23 categories. The validation set contained 337 frame images comprising 3407 instance segmentation counts. The YOLOv8m model was used for training. The loss curve of the training set showed a smooth gradual decrease in loss value as the number of iteration calculations increased. In the training set, the AP values of all 23 categories were above 0.90, with a mAP of 0.99, whereas in the validation set, the mAP of the 23 categories was 0.82. As to individual categories, the AP values for ultrasonic knives, needle holders, forceps, gallbladders, small pieces of gauze, and surgical staplers were 0.96, 0.94, 0.91, 0.91, 0.91, and 0.91, respectively. The model successfully inferred and applied to a 5-minutes video segment of laparoscopic gastroenterostomy suturing. Conclusion: The primary finding of this multicenter study is that computer vision can efficiently, accurately, and in real-time detect organs and instruments in various scenarios of radical laparoscopic gastrectomy for gastric cancer.

Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery

Surgical gesture classification from video and kinematic data

Deep learning prediction of error and skill in robotic prostatectomy suturing

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Gesture Recognition in Robotic Surgery With Multimodal Attention

Road to automating robotic suturing skills assessment: Battling mislabeling of the ground truth

Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery

Multi-stage Suture Detection for Robot Assisted Anastomosis based on Deep Learning

Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication

Vitreoretinal Surgical Instrument Tracking in Three Dimensions Using Deep Learning

Surgical Phase Recognition in Inguinal Hernia Repair—AI-Based Confirmatory Baseline and Exploration of Competitive Models

Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data

Quantification of Robotic Surgeries with Vision-Based Deep Learning

Surgical gesture classification from video data

Towards Augmented Reality-based Suturing in Monocular Laparoscopic Training

[Computer-vision-based artificial intelligence for detection and recognition of instruments and organs during radical laparoscopic gastrectomy for gastric cancer: a multicenter study]

Video-based surgical skill assessment using 3D convolutional neural networks

Surgical Instrument Detection for Laparoscopic Surgery using Deep Learning

Deep Learning Detection of Hand Motion During Microvascular Anastomosis Simulations Performed by Expert Cerebrovascular Neurosurgeons

Gesture Recognition in Robotic Surgery: A Review

Artificial intelligence model for automated surgical instrument detection and counting: an experimental proof-of-concept study