SAMDA: Leveraging SAM on Few-Shot Domain Adaptation for Electronic Microscopy Segmentation

Yiran Wang,Li Xiao

2024-03-12

Abstract:It has been shown that traditional deep learning methods for electronic microscopy segmentation usually suffer from low transferability when samples and annotations are limited, while large-scale vision foundation models are more robust when transferring between different domains but facing sub-optimal improvement under fine-tuning. In this work, we present a new few-shot domain adaptation framework SAMDA, which combines the Segment Anything Model(SAM) with nnUNet in the embedding space to achieve high transferability and accuracy. Specifically, we choose the Unet-based network as the "expert" component to learn segmentation features efficiently and design a SAM-based adaptation module as the "generic" component for domain transfer. By amalgamating the "generic" and "expert" components, we mitigate the modality imbalance in the complex pre-training knowledge inherent to large-scale Vision Foundation models and the challenge of transferability inherent to traditional neural networks. The effectiveness of our model is evaluated on two electron microscopic image datasets with different modalities for mitochondria segmentation, which improves the dice coefficient on the target domain by 6.7%. Also, the SAM-based adaptor performs significantly better with only a single annotated image than the 10-shot domain adaptation on nnUNet. We further verify our model on four MRI datasets from different sources to prove its generalization ability.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in the task of electron microscope image segmentation, traditional deep - learning methods usually show low transfer ability when samples and annotations are limited, while large - scale visual foundation models are more robust when transferring between different fields, but their performance improvement is limited under fine - tuning. Specifically, the paper focuses on how to improve the transfer ability and accuracy of the model from the source domain to the target domain in the case of a small amount of labeled data, especially in the task of mitochondrial segmentation in electron microscope images.

The paper proposes a new few - shot domain adaptation framework (SAMDA), which combines Segment Anything Model (SAM) and nnUNet to achieve high transfer ability and accuracy in the embedding space. By using the UNet - based network as an "expert" component to efficiently learn segmentation features and designing a SAM - based adaptation module as a "general" component for domain transfer, the paper aims to solve the modal imbalance problem in large - scale visual foundation models and the transfer challenges in traditional neural networks.

The main contributions of the paper include:
1. **Proposing the SAMDA framework**: Combining SAM and nnUNet to achieve efficient domain adaptation.
2. **Designing a domain - independent representation learning strategy**: Using perceptual loss to minimize the feature differences between the source domain and the target domain.
3. **Experimental verification**: Experiments were carried out on two electron microscope image data sets and four MRI data sets to verify the effectiveness and generalization ability of the model.

Through these methods, the paper has achieved a significant performance improvement in the task of mitochondrial segmentation of electron microscope images, especially in the few - shot condition, and can significantly outperform the 10 - shot domain - adapted nnUNet using a single labeled image.