Abstract:Background: Endoscopic instrument segmentation is essential for ensuring the safety of robotic-assisted spinal endoscopic surgeries. However, due to the narrow operative region, intricate surrounding tissues, and limited visibility, achieving instrument segmentation within the endoscopic view remains challenging. Purpose: This work aims to devise a method to segment surgical instruments in endoscopic video. By designing an endoscopic image classification model, features of frames before and after the video are extracted to achieve continuous and precise segmentation of instruments in endoscopic videos. Methods: Deep learning techniques serve as the algorithmic core for constructing the convolutional neural network proposed in this study. The method comprises dual stages: image classification and instrument segmentation. MobileViT is employed for image classification, enabling the extraction of key features of different instruments and generating classification results. DeepLabv3+ is utilized for instrument segmentation. By training on distinct instruments separately, corresponding model parameters are obtained. Lastly, a flag caching mechanism along with a blur detection module is designed to effectively utilize the image features in consecutive frames. By incorporating specific parameters into the segmentation model, better segmentation of surgical instruments can be achieved in endoscopic videos. Results: The classification and segmentation models are evaluated on an endoscopic image dataset. In the dataset used for instrument segmentation, the training set consists of 7456 images, the validation set consists of 829 images, and the test set consists of 921 images. In the dataset used for image classification, the training set consists of 2400 images and the validation set consists of 600 images. The image classification model achieves an accuracy of 70% on the validation set. For the segmentation model, experiments are conducted on two common surgical instruments, and the mean Intersection over Union (mIoU) exceeds 98%. Furthermore, the proposed video segmentation method is tested using videos collected during surgeries, validating the effectiveness of the flag caching mechanism and blur detection module. Conclusions: Experimental results on the dataset demonstrate that the dual-stage video processing method excels in performing instrument segmentation tasks under endoscopic conditions. This advancement is significant for enhancing the intelligence level of robotic-assisted spinal endoscopic surgeries.

Deep Learning Assisted Robotic Magnetic Anchored and Guided Endoscope for Real-Time Instrument Tracking

Vision-Based Real-Time Tracking of Surgical Instruments in Robot-Assisted Laparoscopic Surgery

Long term and robust 6DoF motion tracking for highly dynamic stereo endoscopy videos

Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery

A Surgeon Preference-Guided Autonomous Instrument Tracking Method With a Robotic Flexible Endoscope Based on dVRK Platform

3D Position Tracking using On-chip Magnetic Sensing in Image-guided Navigation Bronchoscopy

Development of a flexible endoscopic robot with autonomous tracking control ability using machine vision and deep learning

Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand–Eye Coordination, and Image Definition

A dual-instrument Kalman-based tracker to enhance robustness of microsurgical tools tracking

Multisource Differential Fusion Driven Monocular Endoscope Hybrid 3-D Tracking for Advanced Endoscopic Navigation Surgery

Deep Learning based acoustic measurement approach for robotic applications on orthopedics

SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments

A deep learning framework for real-time 3D model registration in robot-assisted laparoscopic surgery

Semantic SLAM Based on Deep Learning in Endocavity Environment

Tracking endoscopic instruments without localizer: image analysis-based approach

Design, Analysis, and Kinematic Framework of a Long-Range Magnetic Anchored and Guided Endoscope

Closed-Loop Magnetic Manipulation for Robotic Transesophageal Echocardiography

Dual-stage semantic segmentation of endoscopic surgical instruments

Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network

Deep Learning Guided Autonomous Surgery: Guiding Small Needles into Sub-Millimeter Scale Blood Vessels

Motion-Guided Dual-Camera Tracker for Endoscope Tracking and Motion Analysis in a Mechanical Gastric Simulator