Abstract:The accurate segmentation of surgical instruments in endoscopic robotic surgery is a critical challenge due to the intricate and dynamic nature of the surgical environment. Existing segmentation techniques predominantly focus on binary classification, which often falls short in complex scenarios where multiple instruments need to be precisely identified and differentiated. This limitation significantly hampers the effectiveness of computer-assisted surgical systems, where real-time and accurate multi-class segmentation is paramount. In this study, we address these challenges by introducing a novel segmentation framework that leverages pixel-wise contrastive learning to enhance multi-class instrument segmentation. Our approach integrates the strengths of HRNet and DeepLabV3 as backbone networks to effectively capture and embed high-resolution features from endoscopic images. These feature embeddings are stored in dynamically updated dataset-wide queues, enabling the model to mitigate the class imbalance by incorporating representative samples from less prevalent classes during training. This is achieved by combining an enhanced contrastive loss function with pixel-wise cross-entropy loss, which together facilitate robust multi-class differentiation. We validate our approach using the challenging EndoVis 2017 and EndoVis 2018 datasets, where it demonstrated superior performance compared to existing methods. On the EndoVis 2017 test dataset, our HRNet-based method achieved an average mIOU of 0.682 across 10 test subsets, surpassing the official benchmark of 0.542 by a significant margin. Similarly, on the EndoVis 2018 test dataset, our method attained an average mIOU of 0.599, exceeding the official best score of 0.579. These results underscore the efficacy of our approach in significantly improving segmentation accuracy. The enhanced segmentation capability not only advances the state of the art in multi-class instrument segmentation but also has practical implications for improving surgical safety and efficiency in real-world robotic surgery.

Branch Aggregation Attention Network for Robotic Surgical Instrument Segmentation

RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network

LSKANet: Long Strip Kernel Attention Network for Robotic Surgical Scene Segmentation

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision

RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instruments

PaI-Net: A Modified U-Net of Reducing Semantic Gap for Surgical Instrument Segmentation

A parallel network utilizing local features and global representations for segmentation of surgical instruments

Multi-Lateral Branched Network for Tool Segmentation During Robot-Assisted Endovascular Interventions

Accurate instance segmentation of surgical instruments in robotic surgery: model refinement and cross-dataset evaluation

Multi-level feature aggregation network for instrument identification of endoscopic images

Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning

Fast instruments and tissues segmentation of micro-neurosurgical scene using high correlative non-local network

BSANet: Boundary-aware and Scale-Aggregation Networks for CMR Image Segmentation

Semantic Segmentation of Surgical Instruments Based on Enhanced Multi-scale Receptive Field

Unpaired deep adversarial learning for multi-class segmentation of instruments in robot-assisted surgical videos

Real-time instance segmentation of surgical instruments using attention and multi-scale feature fusion

From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments

Hierarchical Attentional Feature Fusion for Surgical Instrument Segmentation

Pixel-wise Contrastive Learning for Multi-class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-wide Sample Queues

Lightweight Deep Neural Network for Real-Time Instrument Semantic Segmentation in Robot Assisted Minimally Invasive Surgery