Abstract:The accurate segmentation of surgical instruments in endoscopic robotic surgery is a critical challenge due to the intricate and dynamic nature of the surgical environment. Existing segmentation techniques predominantly focus on binary classification, which often falls short in complex scenarios where multiple instruments need to be precisely identified and differentiated. This limitation significantly hampers the effectiveness of computer-assisted surgical systems, where real-time and accurate multi-class segmentation is paramount. In this study, we address these challenges by introducing a novel segmentation framework that leverages pixel-wise contrastive learning to enhance multi-class instrument segmentation. Our approach integrates the strengths of HRNet and DeepLabV3 as backbone networks to effectively capture and embed high-resolution features from endoscopic images. These feature embeddings are stored in dynamically updated dataset-wide queues, enabling the model to mitigate the class imbalance by incorporating representative samples from less prevalent classes during training. This is achieved by combining an enhanced contrastive loss function with pixel-wise cross-entropy loss, which together facilitate robust multi-class differentiation. We validate our approach using the challenging EndoVis 2017 and EndoVis 2018 datasets, where it demonstrated superior performance compared to existing methods. On the EndoVis 2017 test dataset, our HRNet-based method achieved an average mIOU of 0.682 across 10 test subsets, surpassing the official benchmark of 0.542 by a significant margin. Similarly, on the EndoVis 2018 test dataset, our method attained an average mIOU of 0.599, exceeding the official best score of 0.579. These results underscore the efficacy of our approach in significantly improving segmentation accuracy. The enhanced segmentation capability not only advances the state of the art in multi-class instrument segmentation but also has practical implications for improving surgical safety and efficiency in real-world robotic surgery.

A Multi-Layer Image Operator Learning Based on Sample Structure for Staff Lines Removal

Image operator learning coupled with CNN classification and its application to staff line removal

Staff line Removal using Generative Adversarial Networks

Research of Numbered Musical Notation Recognition Method

A method for text line detection in natural images

Coordinate Embedding Transformer Model for Optical Music Recognition on Monophonic Scores

Toward a More Complete OMR Solution

Knowledge Discovery in Optical Music Recognition: Enhancing Information Retrieval with Instance Segmentation

A Hybrid Algorithm for the Laser Stripe Centreline Extraction

A detector for page-level handwritten music object recognition based on deep learning

Practical End-to-End Optical Music Recognition for Pianoform Music

An Innovative Approach for Removing Stripe Noise in Infrared Images

Real-Time Optical Music Recognition System for Dulcimer Musical Robot

Domain adaptation for staff-region retrieval of music score images

User Input BBM Match Template Matching Correspondence , Completion and Layer Results Applications Boundary Band Map

TrOMR:Transformer-Based Polyphonic Optical Music Recognition

Pixel-wise Contrastive Learning for Multi-class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-wide Sample Queues

Optimization of Line Extraction Based on an Affine Camera Model

Multi-Constrained Optimization Method of Line Segment Extraction Based on Multi-Scale Image Space

Multi-operator image retargeting with automatic integration of direct and indirect seam carving

Not End-to-End: Explore Multi-Stage Architecture for Online Surgical Phase Recognition