RevSAM2: Prompt SAM2 for Medical Image Segmentation via Reverse-Propagation without Fine-tuning

Yunhao Bai,Boxiang Yun,Zeli Chen,Qinji Yu,Yingda Xia,Yan Wang

2024-11-25

Abstract:The Segment Anything Model 2 (SAM2) has recently demonstrated exceptional performance in zero-shot prompt segmentation for natural images and videos. However, when the propagation mechanism of SAM2 is applied to medical images, it often results in spatial inconsistencies, leading to significantly different segmentation outcomes for very similar images. In this paper, we introduce RevSAM2, a simple yet effective self-correction framework that enables SAM2 to achieve superior performance in unseen 3D medical image segmentation tasks without the need for fine-tuning. Specifically, to segment a 3D query volume using a limited number of support image-label pairs that define a new segmentation task, we propose reverse propagation strategy as a query information selection mechanism. Instead of simply maintaining a first-in-first-out (FIFO) queue of memories to predict query slices sequentially, reverse propagation selects high-quality query information by leveraging support images to evaluate the quality of each predicted query slice mask. The selected high-quality masks are then used as prompts to propagate across the entire query volume, thereby enhancing generalization to unseen tasks. Notably, we are the first to explore the potential of SAM2 in label-efficient medical image segmentation without fine-tuning. Compared to fine-tuning on large labeled datasets, the label-efficient scenario provides a cost-effective alternative for medical segmentation tasks, particularly for rare diseases or when dealing with unseen classes. Experiments on four public datasets demonstrate the superiority of RevSAM2 in scenarios with limited labels, surpassing state-of-the-arts by 12.18% in Dice. The code will be released.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to achieve effective segmentation of 3D medical images by using a small number of annotated 2D support images and their segmentation labels without fine - tuning. Specifically, the author points out that Segment Anything Model 2 (SAM2) performs excellently in zero - sample prompt segmentation of natural images and videos, but when applied to medical images, due to the spatial inconsistency unique to medical images and the changes in the location and appearance of target tissues, the segmentation results are unstable and of poor quality.

To solve this problem, the author proposes the RevSAM2 framework. By introducing the reverse propagation strategy, the performance of SAM2 in the 3D medical image segmentation task is significantly improved without the need for fine - tuning. The following are the main contributions of the paper:

1. **Reverse Propagation Strategy**:
- Traditional forward - propagation methods may lead to a decline in the segmentation quality of subsequent slices due to inaccurate segmentation of early slices.
- The reverse propagation strategy evaluates the quality of each query - slice prediction by back - propagating the predicted query slice and its segmentation mask to the support image. The specific formula is as follows:
\[
\pi_i=\text{AvgDice}(eY_i, Y)
\]
where \(eY_i\) is the predicted segmentation mask of the support image obtained through reverse propagation, and \(Y\) is the true segmentation label of the support image.

2. **Self - Correction Framework**:
- RevSAM2 improves the generalization ability of segmentation by selecting high - quality query slices and propagating them as prompt information throughout the query volume.
- This framework can quickly adapt to new segmentation tasks with only a small number of support images and labels, and is especially suitable for rare diseases or unseen categories.

3. **Experimental Verification**:
- The author conducted experiments on four publicly available multi - organ datasets to verify the superior performance of RevSAM2 in the scenario of limited labels. The experimental results show that RevSAM2 has improved by 12.18% in the Dice coefficient (mDSC) compared to existing methods.

In summary, this paper proposes a novel reverse propagation strategy and self - correction framework, enabling SAM2 to effectively cope with the challenges in medical image segmentation without fine - tuning, especially outstanding in the case of label scarcity.