Abstract:Pre-trained on a large and diverse dataset, the segment anything model (SAM) is the first promptable foundation model in computer vision aiming at object segmentation tasks. In this work, we evaluate SAM for the task of nuclear instance segmentation performance with zero-shot learning and finetuning. We compare SAM with other representative methods in nuclear instance segmentation, especially in the context of model generalisability. To achieve automatic nuclear instance segmentation, we propose using a nuclei detection model to provide bounding boxes or central points of nu-clei as visual prompts for SAM in generating nuclear instance masks from histology images.

What problem does this paper attempt to address?

The paper attempts to address the issue of how to improve the generalization ability of nuclear instance segmentation models across different datasets in digital pathology. Specifically, the authors evaluate the performance of the Segment Anything Model (SAM) on the nuclear instance segmentation task in both zero-shot learning and fine-tuning scenarios, comparing it with other existing methods. The focus of the study is to verify the robustness and generalization ability of the SAM model when handling tissue slice images from different sources, particularly its practical feasibility in clinical applications. ### Main Issues: 1. **Accuracy of Nuclear Instance Segmentation**: Accurately segmenting each cell nucleus in tissue slice images is crucial for subsequent pathological analysis, such as cancer grading and tumor microenvironment analysis. 2. **Model Generalization Ability**: Current machine learning models often perform inconsistently across different datasets, limiting their practicality in clinical applications. Therefore, improving the generalization ability of the model is key. 3. **Comparison of Interactive Segmentation Methods**: By comparing with existing interactive segmentation methods (such as NuClick), the performance of SAM in the nuclear instance segmentation task is evaluated. ### Solutions: - **Using Pre-trained SAM Model**: SAM is a model pre-trained on a large-scale dataset, possessing strong zero-shot generalization ability. - **Combining with Nuclear Detection Model**: To improve segmentation accuracy, the authors propose a two-stage approach. First, YOLOv8 is used for nuclear detection to generate bounding boxes or center points of nuclei, and then this information is input into SAM as visual prompts to generate the final nuclear instance segmentation results. - **Fine-tuning SAM's Mask Decoder**: By fine-tuning SAM's mask decoder, the model's performance on specific datasets is further improved. ### Experimental Results: - **Zero-shot Learning**: Without fine-tuning, SAM shows good segmentation performance when using ground truth bounding boxes as prompts. - **Performance after Fine-tuning**: The fine-tuned SAM, when using nuclear center points as prompts, demonstrates better generalization ability and higher segmentation accuracy compared to NuClick and other methods. ### Conclusion: - **Potential of SAM**: SAM exhibits good generalization ability in the nuclear instance segmentation task, and its performance surpasses existing methods, especially after fine-tuning. - **Prospects for Clinical Application**: Due to its strong generalization ability, SAM has the potential to become a foundational model in digital pathology, providing a reliable tool for nuclear instance segmentation in clinical applications.

On generalisability of segment anything model for nuclear instance segmentation in histology images

UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images

Enhanced Nuclei Segmentation and Classification via Category Descriptors in the SAM Model

NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation

SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model

Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation

Image Segmentation of the Cell Nuclei Staining Using U-net and Segment Anything Model

Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging

Instance segmentation of cells and nuclei from multi-organ cross-protocol microscopic images

Segment anything model for head and neck tumor segmentation with CT, PET and MRI multi-modality images

Pathological Primitive Segmentation Based on Visual Foundation Model with Zero-Shot Mask Generation

Unleashing the Power of Prompt-driven Nucleus Instance Segmentation

Is Segment Anything Model Ready for X-Ray Object Segmentation?

A Dataset and a Technique for Generalized Nuclear Segmentation for Computational Pathology

All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning

Evaluation and improvement of Segment Anything Model for interactive histopathology image segmentation

Accuracy of Segment-Anything Model (SAM) in medical image segmentation tasks

Probability-Based Nuclei Detection and Critical-Region Guided Instance Segmentation.

CellSAM: Advancing Pathologic Image Cell Segmentation via Asymmetric Large‐Scale Vision Model Feature Distillation Aggregation Network

Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation

A dual decoder U-Net-based model for nuclei instance segmentation in hematoxylin and eosin-stained histological images