Path-SAM2: Transfer SAM2 for digital pathology semantic segmentation

Mingya Zhang,Liang Wang,Zhihao Chen,Yiyuan Ge,Xianping Tao

2024-09-04

Abstract:The semantic segmentation task in pathology plays an indispensable role in assisting physicians in determining the condition of tissue lesions. With the proposal of Segment Anything Model (SAM), more and more foundation models have seen rapid development in the field of image segmentation. Recently, SAM2 has garnered widespread attention in both natural image and medical image segmentation. Compared to SAM, it has significantly improved in terms of segmentation accuracy and generalization performance. We compared the foundational models based on SAM and found that their performance in semantic segmentation of pathological images was hardly satisfactory. In this paper, we propose Path-SAM2, which for the first time adapts the SAM2 model to cater to the task of pathological semantic segmentation. We integrate the largest pretrained vision encoder for histopathology (UNI) with the original SAM2 encoder, adding more pathology-based prior knowledge. Additionally, we introduce a learnable Kolmogorov-Arnold Networks (KAN) classification module to replace the manual prompt process. In three adenoma pathological datasets, Path-SAM2 has achieved state-of-the-art performance.This study demonstrates the great potential of adapting SAM2 to pathology image segmentation tasks. We plan to release the code and model weights for this paper at: <a class="link-external link-https" href="https://github.com/simzhangbest/SAM2PATH" rel="external noopener nofollow">this https URL</a>

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main objective of this paper is to address the issue of semantic segmentation in pathology images. Specifically, the paper proposes the Path-SAM2 model, which is the first to apply the SAM2 model to the task of pathology semantic segmentation. It mainly addresses the following issues:

1. **Limitations of existing models in pathology image segmentation**: Although SAM2 performs well in natural image and medical image segmentation, its effectiveness in semantic segmentation of pathology images is suboptimal.

2. **Incorporating more pathological prior knowledge**: By combining the largest pre-trained pathology encoder UNI with the original SAM2 encoder, the model's understanding and segmentation capability in the field of pathology are enhanced.

3. **Improving the manual prompting process**: Introducing a learnable Kolmogorov–Arnold Networks (KAN) classification module to replace the traditional manual prompting method, thereby improving the model's classification accuracy.

4. **Enhancing segmentation performance**: The effectiveness of the Path-SAM2 model was validated on 3 adenoma pathology datasets, achieving the current best performance metrics (DSC and IOU).

In summary, the paper aims to improve the accuracy and robustness of pathology image segmentation tasks by enhancing the SAM2 model and its related components.