Abstract:Diffusion-based personalized visual content generation technologies have achieved significant breakthroughs, allowing for the creation of specific objects by just learning from a few reference photos. However, when misused to fabricate fake news or unsettling content targeting individuals, these technologies could cause considerable societal harm. To address this problem, current methods generate adversarial samples by adversarially maximizing the training loss, thereby disrupting the output of any personalized generation model trained with these samples. However, the existing methods fail to achieve effective defense and maintain stealthiness, as they overlook the intrinsic properties of diffusion models. In this paper, we introduce a novel Dual-Domain Anti-Personalization framework (DDAP). Specifically, we have developed Spatial Perturbation Learning (SPL) by exploiting the fixed and perturbation-sensitive nature of the image encoder in personalized generation. Subsequently, we have designed a Frequency Perturbation Learning (FPL) method that utilizes the characteristics of diffusion models in the frequency domain. The SPL disrupts the overall texture of the generated images, while the FPL focuses on image details. By alternating between these two methods, we construct the DDAP framework, effectively harnessing the strengths of both domains. To further enhance the visual quality of the adversarial samples, we design a localization module to accurately capture attentive areas while ensuring the effectiveness of the attack and avoiding unnecessary disturbances in the background. Extensive experiments on facial benchmarks have shown that the proposed DDAP enhances the disruption of personalized generation models while also maintaining high quality in adversarial samples, making it more effective in protecting privacy in practical applications.

DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models

Adv-BDPM: Adversarial Attack Based on Boundary Diffusion Probability Model.

Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization

Rethinking and Defending Protective Perturbation in Personalized Diffusion Models

DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection

Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model

Toward Robust Imperceptible Perturbation against Unauthorized Text-to-image Diffusion-based Synthesis

DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection

Pixel Is Not A Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models

Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack

Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models

DAP: A Dynamic Adversarial Patch for Evading Person Detectors

Diffusion to Confusion: Naturalistic Adversarial Patch Generation Based on Diffusion Model for Object Detector

Real-time Identity Defenses against Malicious Personalization of Diffusion Models

Information-containing Adversarial Perturbation for Combating Facial Manipulation Systems

Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation

Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World

DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing

Toward Transferable Attack via Adversarial Diffusion in Face Recognition

SimAC: A Simple Anti-Customization Method for Protecting Face Privacy against Text-to-Image Synthesis of Diffusion Models

Adversarial Robust Safeguard for Evading Deep Facial Manipulation