SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathology

Jingwei Zhang,Ke Ma,Saarthak Kapse,Joel Saltz,Maria Vakalopoulou,Prateek Prasanna,Dimitris Samaras

2023-07-13

Abstract:Semantic segmentations of pathological entities have crucial clinical value in computational pathology workflows. Foundation models, such as the Segment Anything Model (SAM), have been recently proposed for universal use in segmentation tasks. SAM shows remarkable promise in instance segmentation on natural images. However, the applicability of SAM to computational pathology tasks is limited due to the following factors: (1) lack of comprehensive pathology datasets used in SAM training and (2) the design of SAM is not inherently optimized for semantic segmentation tasks. In this work, we adapt SAM for semantic segmentation by introducing trainable class prompts, followed by further enhancements through the incorporation of a pathology encoder, specifically a pathology foundation model. Our framework, SAM-Path enhances SAM's ability to conduct semantic segmentation in digital pathology without human input prompts. Through experiments on two public pathology datasets, the BCSS and the CRAG datasets, we demonstrate that the fine-tuning with trainable class prompts outperforms vanilla SAM with manual prompts and post-processing by 27.52% in Dice score and 71.63% in IOU. On these two datasets, the proposed additional pathology foundation model further achieves a relative improvement of 5.07% to 5.12% in Dice score and 4.50% to 8.48% in IOU.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the task of semantic segmentation in digital pathology. Specifically, basic models such as the Segment Anything Model (SAM) have shown significant potential in instance segmentation of natural images, but there are two main limitations when directly applied to digital pathology tasks:

1. **Lack of comprehensive pathological datasets**: The datasets used in SAM training lack diverse pathological images, which limits its effective application in digital pathology tasks.
2. **Limitations in design**: The design of SAM requires manual input prompts (such as points, bounding boxes, or text), and there is no component specifically for semantic classification. Therefore, it does not directly support the semantic segmentation task, while semantic segmentation is crucial for identifying and distinguishing different tissue structures, cell types, and sub - cellular components in digital pathology.

To solve these problems, the authors propose a new method adapted to semantic segmentation in digital pathology - SAM - Path. This method enhances the capabilities of SAM by introducing trainable class prompts and a pathological encoder, enabling it to perform semantic segmentation without manual prompts. Specific improvements include:

- **Trainable class prompts**: By introducing trainable class prompts, the model can recognize semantic information of multiple classes.
- **Pathological encoder**: Add a pathological base model as an additional feature encoder to provide domain - specific knowledge.

Through experiments on two publicly available pathological datasets (BCSS and CRAG), the authors demonstrate a significant performance improvement of this method compared to the original SAM. Specifically:

- On the CRAG dataset, SAM - Path using trainable prompts has a relative improvement of 27.52% in Dice score and 71.63% in IOU compared to the original SAM with manual prompts.
- After further adding the pathological encoder, the Dice score and IOU are relatively improved by 5.07% - 5.12% on the BCSS dataset and 4.50% - 8.48% on the CRAG dataset respectively.

These results indicate that by introducing trainable class prompts and a pathological encoder, SAM - Path can significantly improve the performance of semantic segmentation in digital pathology.