Abstract:Adversarial attacks, particularly \textbf{targeted} transfer-based attacks, can be used to assess the adversarial robustness of large visual-language models (VLMs), allowing for a more thorough examination of potential security flaws before deployment. However, previous transfer-based adversarial attacks incur high costs due to high iteration counts and complex method structure. Furthermore, due to the unnaturalness of adversarial semantics, the generated adversarial examples have low transferability. These issues limit the utility of existing methods for assessing robustness. To address these issues, we propose AdvDiffVLM, which uses diffusion models to generate natural, unrestricted and targeted adversarial examples via score matching. Specifically, AdvDiffVLM uses Adaptive Ensemble Gradient Estimation to modify the score during the diffusion model's reverse generation process, ensuring that the produced adversarial examples have natural adversarial targeted semantics, which improves their transferability. Simultaneously, to improve the quality of adversarial examples, we use the GradCAM-guided Mask method to disperse adversarial semantics throughout the image rather than concentrating them in a single area. Finally, AdvDiffVLM embeds more target semantics into adversarial examples after multiple iterations. Experimental results show that our method generates adversarial examples 5x to 10x faster than state-of-the-art transfer-based adversarial attacks while maintaining higher quality adversarial examples. Furthermore, compared to previous transfer-based adversarial attacks, the adversarial examples generated by our method have better transferability. Notably, AdvDiffVLM can successfully attack a variety of commercial VLMs in a black-box environment, including GPT-4V.

LESEP: Boosting Adversarial Transferability Via Latent Encoding and Semantic Embedding Perturbations

Boosting the Targeted Transferability of Adversarial Examples via Salient Region & Weighted Feature Drop

Understanding and Enhancing the Transferability of Adversarial Examples

Enhancing adversarial attacks with resize-invariant and logical ensemble

Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models

Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability

Towards Transferable Unrestricted Adversarial Examples with Minimum Changes

Typography Leads Semantic Diversifying: Amplifying Adversarial Transferability across Multimodal Large Language Models

Strengthening transferability of adversarial examples by adaptive inertia and amplitude spectrum dropout

Improving Adversarial Transferability by Stable Diffusion

PEAS: A Strategy for Crafting Transferable Adversarial Examples

Bag of Tricks to Boost Adversarial Transferability

Boosting the Transferability of Video Adversarial Examples Via Temporal Translation.

LRS: Enhancing Adversarial Transferability through Lipschitz Regularized Surrogate

Diffusion Models for Imperceptible and Transferable Adversarial Attack

Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks

Rethinking the Backward Propagation for Adversarial Transferability

LFAA: Crafting Transferable Targeted Adversarial Examples with Low-Frequency Perturbations

Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model

Demiguise Attack: Crafting Invisible Semantic Adversarial Perturbations with Perceptual Similarity

Boosting the Transferability of Adversarial Examples via Local Mixup and Adaptive Step Size