Abstract:The Segment Anything Model (SAM) is a cornerstone of image segmentation, demonstrating exceptional performance across various applications, particularly in autonomous driving and medical imaging, where precise segmentation is crucial. However, SAM is vulnerable to adversarial attacks that can significantly impair its functionality through minor input perturbations. Traditional techniques, such as FGSM and PGD, are often ineffective in segmentation tasks due to their reliance on global perturbations that overlook spatial nuances. Recent methods like Attack-SAM-K and UAD have begun to address these challenges, but they frequently depend on external cues and do not fully leverage the structural interdependencies within segmentation processes. This limitation underscores the need for a novel adversarial strategy that exploits the unique characteristics of segmentation tasks. In response, we introduce the Region-Guided Attack (RGA), designed specifically for SAM. RGA utilizes a Region-Guided Map (RGM) to manipulate segmented regions, enabling targeted perturbations that fragment large segments and expand smaller ones, resulting in erroneous outputs from SAM. Our experiments demonstrate that RGA achieves high success rates in both white-box and black-box scenarios, emphasizing the need for robust defenses against such sophisticated attacks. RGA not only reveals SAM's vulnerabilities but also lays the groundwork for developing more resilient defenses against adversarial threats in image segmentation.

What problem does this paper attempt to address?

This paper attempts to address the vulnerability of the Segment Anything Model (SAM) when facing adversarial attacks. Specifically, although SAM performs excellently in image segmentation tasks, its functionality may be significantly impaired under the influence of minor input perturbations (i.e., adversarial attacks). Traditional adversarial attack methods (such as FGSM and PGD) are ineffective in segmentation tasks because they rely on global perturbations and overlook spatial details. Although recent methods (such as Attack - SAM - K and UAD) have begun to address these challenges, they still rely on external cues and do not fully utilize the structural dependencies in the segmentation process. Therefore, this paper proposes a new adversarial attack strategy - Region - Guided Attack (RGA), which aims to specifically target the weaknesses of SAM for attack. RGA manipulates the segmentation area by introducing the Region - Guided Map (RGM) to split large areas and merge small areas, thereby causing SAM to output errors. The main contributions of the paper include: 1. **Region - Guided Map (RGM) for Adversarial Guidance**: RGA uses RGM to directly guide the generation of adversarial samples. By defining how to change the segmentation of SAM (for example, dividing a large area into smaller parts or merging small areas into a larger part), it effectively guides the perturbation to maximize the impact on the segmentation quality. 2. **Enhanced Attack Success and Transferability**: By utilizing RGM, RGA achieves a higher attack success rate and better transferability. The generation of adversarial samples is influenced by the explicit goals of the segmentation output, systematically guiding the perturbation, resulting in more successful and more transferable attacks on the SAM model. 3. **Independent of External Prompts**: Unlike many existing methods that rely on specific cues to guide attacks, RGA is independent of external prompts, making the adversarial process more simplified and widely applicable. This independence ensures that RGA can be applied in the absence or unpredictability of prompts. 4. **Insights into SAM’s Segmentation Vulnerabilities**: RGA reveals specific vulnerabilities of SAM by focusing on area operations rather than global input perturbations. The study found that changing the size and boundaries of the segmentation area can significantly reduce the segmentation performance of SAM, providing valuable insights for designing more powerful segmentation models. In conclusion, this paper not only reveals the vulnerabilities existing in advanced segmentation models such as SAM but also lays the foundation for developing more powerful defense measures against such targeted attacks.

Region-Guided Attack on the Segment Anything Model (SAM)

Practical Region-level Attack against Segment Anything Models

Black-box Targeted Adversarial Attack on Segment Anything (SAM)

Attack-SAM: Towards Attacking Segment Anything Model With Adversarial Examples

DarkSAM: Fooling Segment Anything Model to Segment Nothing

On the Robustness of Segment Anything

SAM Meets UAP: Attacking Segment Anything Model With Universal Adversarial Perturbation

SAM Fails to Segment Anything? – SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More

Cross-Point Adversarial Attack Based on Feature Neighborhood Disruption Against Segment Anything Model

Adversarial Attacks on Video Object Segmentation with Hard Region Discovery

BadSAM: Exploring Security Vulnerabilities of SAM via Backdoor Attacks

Transferable Adversarial Attacks on SAM and Its Downstream Models

SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness

An Empirical Study on the Robustness of the Segment Anything Model (SAM)

Segment-Anything Models Achieve Zero-shot Robustness in Autonomous Driving

SAM-Adapter: Adapting Segment Anything in Underperformed Scenes

ASAM: Boosting Segment Anything Model with Adversarial Tuning

Stealthy Adversarial Examples for Semantic Segmentation in Remote Sensing

AGSAM: Agent-Guided Segment Anything Model for Automatic Segmentation in Few-Shot Scenarios

RobustSAM: Segment Anything Robustly on Degraded Images

Evaluating the Efficacy of Segment Anything Model for Delineating Agriculture and Urban Green Spaces in Multiresolution Aerial and Spaceborne Remote Sensing Images