Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack

Xide Xu,Muhammad Atif Butt,Sandesh Kamath,Bogdan Raducanu

2024-11-25

Abstract:The growing demand for customized visual content has led to the rise of personalized text-to-image (T2I) diffusion models. Despite their remarkable potential, they pose significant privacy risk when misused for malicious purposes. In this paper, we propose a novel and efficient adversarial attack method, Concept Protection by Selective Attention Manipulation (CoPSAM) which targets only the cross-attention layers of a T2I diffusion model. For this purpose, we carefully construct an imperceptible noise to be added to clean samples to get their adversarial counterparts. This is obtained during the fine-tuning process by maximizing the discrepancy between the corresponding cross-attention maps of the user-specific token and the class-specific token, respectively. Experimental validation on a subset of CelebA-HQ face images dataset demonstrates that our approach outperforms existing methods. Besides this, our method presents two important advantages derived from the qualitative evaluation: (i) we obtain better protection results for lower noise levels than our competitors; and (ii) we protect the content from unauthorized use thereby protecting the individual's identity from potential misuse.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

This paper attempts to address the privacy - protection issues in personalized text - to - image (T2I) diffusion models. Although these models have significant potential in generating customized visual content, they also bring the risk of being maliciously exploited, such as being used to generate deceptive images ("deepfakes"). The paper proposes a new adversarial attack method - Concept Protection through Selective Attention Manipulation (CoPSAM), specifically targeting the cross - attention layers in T2I diffusion models. Through this method, the authors aim to protect personal privacy by minimizing the similarity between user - specific tokens and category - specific tokens, while maintaining the identity characteristics of individuals from being misused. Specifically, CoPSAM constructs an almost imperceptible noise by maximizing the difference between the cross - attention maps corresponding to user - specific tokens and category - specific tokens during the training process, and adds it to clean samples to obtain adversarial samples. Experimental verification shows that CoPSAM can achieve better protection effects than existing methods at a lower noise level, and can also effectively prevent the unauthorized use of personalized content, thereby protecting personal identities from potential misuse.

Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack

Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization

Targeted Attack Improves Protection against Unauthorized Diffusion Customization

Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models

Toward Robust Imperceptible Perturbation against Unauthorized Text-to-image Diffusion-based Synthesis

Rethinking and Defending Protective Perturbation in Personalized Diffusion Models

Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model

DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models

DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing

DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection

Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models

Protecting by Attacking: A Personal Information Protecting Method with Cross-Modal Adversarial Examples

SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models

TAFIM: Targeted Adversarial Attacks against Facial Image Manipulations

Diffusion Models for Imperceptible and Transferable Adversarial Attack

Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks

Exploiting vulnerabilities of deep neural networks for privacy protection

Cheating Suffix: Targeted Attack to Text-To-Image Diffusion Models with Multi-Modal Priors

Adversarial Examples for Preventing Diffusion Models from Malicious Image Edition

Adversarial Robust Safeguard for Evading Deep Facial Manipulation

Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey