Abstract:Foundational vision models, such as the Segment Anything Model (SAM), have achieved significant breakthroughs through extensive pre-training on large-scale visual datasets. Despite their general success, these models may fall short in specialized tasks with limited data, and fine-tuning such large-scale models is often not feasible. Current strategies involve incorporating adaptors into the pre-trained SAM to facilitate downstream task performance with minimal model adjustment. However, these strategies can be hampered by suboptimal learning approaches for the adaptors. In this paper, we introduce a novel Multi-scale Contrastive Adaptor learning method named MCA-SAM, which enhances adaptor performance through a meticulously designed contrastive learning framework at both token and sample levels. Our Token-level Contrastive adaptor (TC-adaptor) focuses on refining local representations by improving the discriminability of patch tokens, while the Sample-level Contrastive adaptor (SC-adaptor) amplifies global understanding across different samples. Together, these adaptors synergistically enhance feature comparison within and across samples, bolstering the model's representational strength and its ability to adapt to new tasks. Empirical results demonstrate that MCA-SAM sets new benchmarks, outperforming existing methods in three challenging domains: camouflage object detection, shadow segmentation, and polyp segmentation. Specifically, MCA-SAM exhibits substantial relative performance enhancements, achieving a 20.0% improvement in MAE on the COD10K dataset, a 6.0% improvement in MAE on the CAMO dataset, a 15.4% improvement in BER on the ISTD dataset, and a 7.9% improvement in mDice on the Kvasir-SEG dataset.

MST: Adaptive Multi-Scale Tokens Guided Interactive Segmentation

Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic Segmentation

Adaptive fusion with multi-scale features for interactive image segmentation

Token Sparsification for Faster Medical Image Segmentation

FocalClick: Towards Practical Interactive Image Segmentation.

Enhanced Multi-Scale Feature Adaptive Fusion Sparse Convolutional Network for Large-Scale Scenes Semantic Segmentation

Interactive segmentation based on multiscale feature cascading

MT-SCnet: multi-scale token divided and spatial-channel fusion transformer network for microscopic hyperspectral image segmentation

Learning from Exemplars for Interactive Image Segmentation

Adaptive Interactive Segmentation for Multimodal Medical Imaging via Selection Engine

Scale Disparity of Instances in Interactive Point Cloud Segmentation

Multi-granularity Interaction Simulation for Unsupervised Interactive Segmentation

SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation

Multi-scale Contrastive Adaptor Learning for Segmenting Anything in Underperformed Scenes

Multi-Scale Representations by Varying Window Attention for Semantic Segmentation

Multi-Scale And Token Mergence: Make Your ViT More Efficient

Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation

Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation

Multi-Scale Attentive Aggregation for LiDAR Point Cloud Segmentation

Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation