Segment Anything Model-guided Collaborative Learning Network for Scribble-supervised Polyp Segmentation

Yiming Zhao,Tao Zhou,Yunqi Gu,Yi Zhou,Yizhe Zhang,Ye Wu,Huazhu Fu

2023-12-01

Abstract:Polyp segmentation plays a vital role in accurately locating polyps at an early stage, which holds significant clinical importance for the prevention of colorectal cancer. Various polyp segmentation methods have been developed using fully-supervised deep learning techniques. However, pixel-wise annotation for polyp images by physicians during the diagnosis is both time-consuming and expensive. Moreover, visual foundation models such as the Segment Anything Model (SAM) have shown remarkable performance. Nevertheless, directly applying SAM to medical segmentation may not produce satisfactory results due to the inherent absence of medical knowledge. In this paper, we propose a novel SAM-guided Collaborative Learning Network (SAM-CLNet) for scribble-supervised polyp segmentation, enabling a collaborative learning process between our segmentation network and SAM to boost the model performance. Specifically, we first propose a Cross-level Enhancement and Aggregation Network (CEA-Net) for weakly-supervised polyp segmentation. Within CEA-Net, we propose a Cross-level Enhancement Module (CEM) that integrates the adjacent features to enhance the representation capabilities of different resolution features. Additionally, a Feature Aggregation Module (FAM) is employed to capture richer features across multiple levels. Moreover, we present a box-augmentation strategy that combines the segmentation maps generated by CEA-Net with scribble annotations to create more precise prompts. These prompts are then fed into SAM, generating segmentation SAM-guided masks, which can provide additional supervision to train CEA-Net effectively. Furthermore, we present an Image-level Filtering Mechanism to filter out unreliable SAM-guided masks. Extensive experimental results show that our SAM-CLNet outperforms state-of-the-art weakly-supervised segmentation methods.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the challenges in polyp segmentation, particularly the issues encountered when performing weakly supervised segmentation with limited annotated data. Specifically:

1. **Reducing Dependence on Fully Supervised Methods**: Traditional deep learning-based polyp segmentation methods primarily rely on fully supervised learning, which requires a large amount of pixel-level annotations. These annotations are time-consuming and costly.
2. **Introducing SAM (Segment Anything Model) Enhanced Collaborative Learning Network**: The authors propose a new SAM-guided collaborative learning network (SAM-CLNet), which combines segmentation masks generated by SAM to enhance model performance. In this way, the model can learn more details from limited annotated information, improving segmentation accuracy.
3. **Developing New Weak Supervision Methods**: For the polyp segmentation task, a novel weak supervision method—Cross-level Enhancement and Aggregation Network (CEA-Net)—is proposed. This network integrates multi-level features to improve segmentation results.
4. **Improving Model Robustness**: To further enhance model accuracy, the paper also proposes a box enhancement strategy and an image-level filtering mechanism to generate more reliable SAM-guided masks.

Through the above methods, this research significantly improves polyp segmentation performance with limited annotations and surpasses existing weakly supervised segmentation methods in experimental results.