Abstract:In surgical procedures, correct instrument counting is essential. Instance segmentation is a location method that locates not only an object's bounding box but also each pixel's specific details. However, obtaining mask-level annotations is labor-intensive in instance segmentation. To address this issue, we propose a novel yet effective weakly-supervised surgical instrument instance segmentation approach, named Point-based Weakly-supervised Instance Segmentation (PWISeg). PWISeg adopts an FCN-based architecture with point-to-box and point-to-mask branches to model the relationships between feature points and bounding boxes, as well as feature points and segmentation masks on FPN, accomplishing instrument detection and segmentation jointly in a single model. Since mask level annotations are hard to available in the real world, for point-to-mask training, we introduce an unsupervised projection loss, utilizing the projected relation between predicted masks and bboxes as supervision signal. On the other hand, we annotate a few pixels as the key pixel for each instrument. Based on this, we further propose a key pixel association loss and a key pixel distribution loss, driving the point-to-mask branch to generate more accurate segmentation predictions. To comprehensively evaluate this task, we unveil a novel surgical instrument dataset with manual annotations, setting up a benchmark for further research. Our comprehensive research trial validated the superior performance of our PWISeg. The results show that the accuracy of surgical instrument segmentation is improved, surpassing most methods of instance segmentation via weakly supervised bounding boxes. This improvement is consistently observed in our proposed dataset and when applied to the public HOSPI-Tools dataset.

What problem does this paper attempt to address?

The paper primarily addresses the problem of accurately identifying and locating surgical instruments during operations, especially in scenarios where instruments are densely stacked and occluded. To improve the accuracy of surgical instrument recognition, particularly in these complex scenes, the paper proposes a new method called Point-based Weakly-supervised Instance Segmentation (PWISeg). ### Main Issues 1. **Importance of Accurate Counting of Surgical Instruments**: Accurately counting all tools used during surgery is crucial because any tool left inside the patient can cause infections or bodily harm. 2. **Challenges Faced by Existing Technologies**: A major challenge for computer vision-based methods in real operating room environments is the dense stacking and occlusion of surgical instruments, making it difficult for existing detection methods to accurately locate the instruments. 3. **Limitations of Fully Supervised Instance Segmentation**: Fully supervised instance segmentation requires resource-intensive mask-level annotations, which are often difficult to obtain in practice. ### Solution To address the above issues, the authors propose a weakly supervised learning method, PWISeg, which has the following main features: - **Architecture Design**: It uses FCN (Fully Convolutional Network) as the base architecture and combines point-to-box and point-to-mask branches to model the relationship between feature points and bounding boxes as well as segmentation masks. - **Training Strategy**: - In point-to-box training, Focal Loss is used to evaluate the consistency between the model's predicted categories and actual labels as a supervision signal; IOU loss is also used to assess the match between predicted and actual bounding boxes. - For point-to-mask training, in the absence of mask-level annotations, an unsupervised projection loss is introduced, utilizing the projection relationship between the predicted mask and bounding box as a supervision signal. - Several key pixels are annotated for each instrument, and based on this, key pixel association loss and key pixel distribution loss are proposed to drive the point-to-mask branch to generate more accurate segmentation predictions. ### Experimental Results - The authors conducted experimental validation on a newly proposed surgical instrument dataset, which includes professionally annotated key points and bounding boxes, aiming to accelerate research and development in the field of surgical instrument segmentation. - On this dataset, PWISeg achieved 23.9% mean Average Precision (mAP) and also achieved 30.6% mAP on the public HOSPI-Tools dataset, demonstrating its effectiveness. Through the above methods, PWISeg not only simplifies the annotation process but also significantly improves the segmentation accuracy of surgical instruments, showing potential application value in real-time surgical assistance and enhancing operational efficiency in the medical field.

PWISeg: Point-based Weakly-supervised Instance Segmentation for Surgical Instruments

Weakly Supervised Instance Segmentation Using Multi-Prior Fusion.

WeakSurg: Weakly supervised surgical instrument segmentation using temporal equivariance and semantic continuity

Accurate instance segmentation of surgical instruments in robotic surgery: model refinement and cross-dataset evaluation

PaI-Net: A Modified U-Net of Reducing Semantic Gap for Surgical Instrument Segmentation

Pixel-wise Contrastive Learning for Multi-class Instrument Segmentation in Endoscopic Robotic Surgery Videos Using Dataset-wide Sample Queues

Many Birds, One Stone: Medical Image Segmentation with Multiple Partially Labeled Datasets

From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments

Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery

SINet: A hybrid deep CNN model for real-time detection and segmentation of surgical instruments

Weakly Supervised Pose Estimation of Surgical Instrument from a Single Endoscopic Image

ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding

Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View

A new weakly supervised strategy for surgical tool detection

RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network

Semantic Segmentation of Surgical Instruments Based on Enhanced Multi-scale Receptive Field

SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation

A parallel network utilizing local features and global representations for segmentation of surgical instruments

A Point-Neighborhood Learning Framework for Nasal Endoscope Image Segmentation

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion