Abstract:Recent advancements in diffusion models revolutionize image generation but pose risks of misuse, such as replicating artworks or generating deepfakes. Existing image protection methods, though effective, struggle to balance protection efficacy, invisibility, and latency, thus limiting practical use. We introduce perturbation pre-training to reduce latency and propose a mixture-of-perturbations approach that dynamically adapts to input images to minimize performance degradation. Our novel training strategy computes protection loss across multiple VAE feature spaces, while adaptive targeted protection at inference enhances robustness and invisibility. Experiments show comparable protection performance with improved invisibility and drastically reduced inference time. The code and demo are available at \url{<a class="link-external link-https" href="https://webtoon.github.io/impasto" rel="external noopener nofollow">this https URL</a>}

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem of the misuse of diffusion models in image generation, especially malicious behaviors such as copying artworks or generating deepfakes. Although existing image protection methods are effective, it is difficult to strike a balance between protection effectiveness, invisibility and latency, which limits their practical applications. Specifically, the paper mainly focuses on the following three problems: 1. **Protection Efficacy**: Image protection methods should be able to effectively resist personalized attacks based on diffusion models. Many studies maximize this aspect by analyzing the internal dynamics of diffusion models. 2. **Invisibility**: Adversarial perturbations will inevitably leave visible traces on the image, but minimizing these traces is crucial in commercial services. Especially in images with smooth textures such as cartoons or illustrations, the task of preventing imitation often leaves more obvious traces. 3. **Latency**: All existing methods update perturbations through iterative optimization in the inference stage, and this process is very time - consuming. For example, protecting a 512×512 - pixel image may take 5 - 120 minutes on a CPU and 7 - 200 seconds on a high - performance A100 GPU. This long - term processing hinders ordinary users from using image protection technologies, leaving them still at risk of the misuse of generation models. To solve these problems, the paper proposes a new protection framework - FastProtect, which specifically focuses on reducing latency while maintaining protection effectiveness and invisibility. FastProtect realizes real - time protection through pre - trained perturbations and an adaptive inference scheme, and verifies its effectiveness and efficiency in multiple scenarios. ### Summary The core problem of the paper is to develop a protection method that can not only effectively protect images from the misuse of diffusion models, but also significantly reduce processing time and be as invisible as possible. FastProtect successfully solves the above challenges by introducing techniques such as Mixture - of - Perturbations (MoP), Multi - Layer Protection Loss and adaptive target protection.

Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models

Imperceptible Protection against Style Imitation from Diffusion Models

Toward effective protection against diffusion based mimicry through score distillation

Toward Robust Imperceptible Perturbation against Unauthorized Text-to-image Diffusion-based Synthesis

Disrupting Style Mimicry Attacks on Video Imagery

IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI

Pixel Is Not A Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models

Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models

DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing

Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models

Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation

Raising the Cost of Malicious AI-Powered Image Editing

Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack

Attack as Defense: Run-time Backdoor Implantation for Image Content Protection

MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Diffusion-based Synthesis via Meta-learning

Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?

Rethinking and Defending Protective Perturbation in Personalized Diffusion Models

Adversarial Robust Safeguard for Evading Deep Facial Manipulation

Only My Model On My Data: A Privacy Preserving Approach Protecting one Model and Deceiving Unauthorized Black-Box Models

Anti-DreamBooth: Protecting users from personalized text-to-image synthesis

Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model