Evaluation Study on SAM 2 for Class-agnostic Instance-level Segmentation

Jialun Pei,Zhangjun Zhou,Tiantian Zhang

2024-10-02

Abstract:Segment Anything Model (SAM) has demonstrated powerful zero-shot segmentation performance in natural scenes. The recently released Segment Anything Model 2 (SAM2) has further heightened researchers' expectations towards image segmentation capabilities. To evaluate the performance of SAM2 on class-agnostic instance-level segmentation tasks, we adopt different prompt strategies for SAM2 to cope with instance-level tasks for three relevant scenarios: Salient Instance Segmentation (SIS), Camouflaged Instance Segmentation (CIS), and Shadow Instance Detection (SID). In addition, to further explore the effectiveness of SAM2 in segmenting granular object structures, we also conduct detailed tests on the high-resolution Dichotomous Image Segmentation (DIS) benchmark to assess the fine-grained segmentation capability. Qualitative and quantitative experimental results indicate that the performance of SAM2 varies significantly across different scenarios. Besides, SAM2 is not particularly sensitive to segmenting high-resolution fine details. We hope this technique report can drive the emergence of SAM2-based adapters, aiming to enhance the performance ceiling of large vision models on class-agnostic instance segmentation tasks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the performance of Segment Anything Model 2 (SAM2) in the class - agnostic instance - level segmentation tasks. Specifically, the researchers focused on the following aspects:

1. **Performance evaluation in different scenarios**: The researchers adopted different prompting strategies to test the instance - level task performance of SAM2 in three related scenarios, namely Salient Instance Segmentation (SIS), Camouflaged Instance Segmentation (CIS), and Shadow Instance Detection (SID). In addition, in order to further explore the effectiveness of SAM2 in segmenting fine - grained target structures, detailed tests were also carried out on the High - Resolution Dichotomous Image Segmentation (DIS) benchmark.

2. **Evaluation of fine - grained segmentation ability**: Through the tests on the DIS benchmark, the ability of SAM2 to perform fine - grained segmentation on complex object structures was evaluated.

3. **Comparison with existing models**: The researchers compared SAM2 with SAM and task - specific models on multiple benchmarks to evaluate its performance on different tasks.

4. **Exploration of zero - shot segmentation ability**: The paper explored whether SAM2 can effectively handle various instance - level segmentation tasks without additional training.

Through these evaluations, the researchers hope to reveal the advantages and limitations of SAM2 in different scenarios, and provide guidance for the future development of adapters based on SAM2, thereby increasing the performance ceiling of large - scale visual models in class - agnostic instance - level segmentation tasks.