CycleSAM: One-Shot Surgical Scene Segmentation using Cycle-Consistent Feature Matching to Prompt SAM

Aditya Murali,Pietro Mascagni,Didier Mutter,Nicolas Padoy

2024-07-09

Abstract:The recently introduced Segment-Anything Model (SAM) has the potential to greatly accelerate the development of segmentation models. However, directly applying SAM to surgical images has key limitations including (1) the requirement of image-specific prompts at test-time, thereby preventing fully automated segmentation, and (2) ineffectiveness due to substantial domain gap between natural and surgical images. In this work, we propose CycleSAM, an approach for one-shot surgical scene segmentation that uses the training image-mask pair at test-time to automatically identify points in the test images that correspond to each object class, which can then be used to prompt SAM to produce object masks. To produce high-fidelity matches, we introduce a novel spatial cycle-consistency constraint that enforces point proposals in the test image to rematch to points within the object foreground region in the training image. Then, to address the domain gap, rather than directly using the visual features from SAM, we employ a ResNet50 encoder pretrained on surgical images in a self-supervised fashion, thereby maintaining high label-efficiency. We evaluate CycleSAM for one-shot segmentation on two diverse surgical semantic segmentation datasets, comprehensively outperforming baseline approaches and reaching up to 50% of fully-supervised performance.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper proposes a new method called CycleSAM to solve the surgical scene segmentation problem. The existing Segment-Anything Model (SAM) performs well in natural image segmentation, but it has two main limitations on surgical images: (1) it requires image-specific hints during testing, making fully automated segmentation impossible; (2) the significant domain differences between natural images and surgical images lead to unsatisfactory results. CycleSAM aims to achieve one-shot surgical scene segmentation by training image-mask pairs to automatically identify key points corresponding to each object category during testing, and then use these points to guide SAM in generating object masks.

To improve matching accuracy, the paper introduces a spatial cycle consistency constraint, which forces the key points in the test image to re-match to points within the foreground region of the object in the training image. In addition, they use a pre-trained ResNet50 encoder on surgical images to handle feature matching, ensuring high label efficiency. To address domain gaps, they enhance the feature extractor of SAM by leveraging the self-supervised feature extractors MoCov2 and DINO.

CycleSAM is evaluated on two different surgical semantic segmentation datasets, surpassing baseline methods and achieving approximately 50% of fully supervised performance, significantly improving surgical scene segmentation performance under high label efficiency. The main contributions of the paper include: (1) proposing the CycleSAM method, which is based on feature matching for one-shot adaptation of SAM; (2) introducing the CycleSelect module to improve key point matching; (3) comprehensively surpassing current SAM adaptation methods in high label efficiency surgical scene segmentation.