Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging

Ruining Deng,Can Cui,Quan Liu,Tianyuan Yao,Lucas W. Remedios,Shunxing Bao,Bennett A. Landman,Lee E. Wheless,Lori A. Coburn,Keith T. Wilson,Yaohong Wang,Shilin Zhao,Agnes B. Fogo,Haichun Yang,Yucheng Tang,Yuankai Huo

2023-04-09

Abstract:The segment anything model (SAM) was released as a foundation model for image segmentation. The promptable segmentation model was trained by over 1 billion masks on 11M licensed and privacy-respecting images. The model supports zero-shot image segmentation with various segmentation prompts (e.g., points, boxes, masks). It makes the SAM attractive for medical image analysis, especially for digital pathology where the training data are rare. In this study, we evaluate the zero-shot segmentation performance of SAM model on representative segmentation tasks on whole slide imaging (WSI), including (1) tumor segmentation, (2) non-tumor tissue segmentation, (3) cell nuclei segmentation. Core Results: The results suggest that the zero-shot SAM model achieves remarkable segmentation performance for large connected objects. However, it does not consistently achieve satisfying performance for dense instance object segmentation, even with 20 prompts (clicks/boxes) on each image. We also summarized the identified limitations for digital pathology: (1) image resolution, (2) multiple scales, (3) prompt selection, and (4) model fine-tuning. In the future, the few-shot fine-tuning with images from downstream pathological segmentation tasks might help the model to achieve better performance in dense object segmentation.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to use zero - shot segmentation techniques in digital pathology to handle different tasks in whole slide images (WSI), such as tumor segmentation, non - tumor tissue segmentation, and nuclear segmentation. Specifically, the researchers evaluated the zero - shot performance of the "Segment Anything Model" (SAM) on these tasks. SAM is a base model, trained on 11 million licensed and privacy - respecting images with over 1 billion masks, supporting zero - shot image segmentation under multiple segmentation cues (e.g., points, boxes, masks). The main purpose of this study is to explore the application potential of SAM in medical image analysis, especially in digital pathology, where labeled training data is very scarce and expensive.

The study found that SAM performs well in the segmentation of large connected objects, but inconsistently in the segmentation of dense instance objects, even when 20 cues (clicks or boxes) are provided per image. In addition, the study also summarized the limitations identified in digital pathology, including image resolution, multi - scale problems, cue selection, and model fine - tuning. Future research may need to be fine - tuned with a small number of images from downstream pathological segmentation tasks to help the model achieve better performance in dense object segmentation.