Abstract:Recent advances in large-scale text-to-image (T2I) diffusion models have enabled a variety of downstream applications, including style customization, subject-driven personalization, and conditional generation. As T2I models require extensive data and computational resources for training, they constitute highly valued intellectual property (IP) for their legitimate owners, yet making them incentive targets for unauthorized fine-tuning by adversaries seeking to leverage these models for customized, usually profitable applications. Existing IP protection methods for diffusion models generally involve embedding watermark patterns and then verifying ownership through generated outputs examination, or inspecting the model's feature space. However, these techniques are inherently ineffective in practical scenarios when the watermarked model undergoes fine-tuning, and the feature space is inaccessible during verification ((i.e., black-box setting). The model is prone to forgetting the previously learned watermark knowledge when it adapts to a new task. To address this challenge, we propose SleeperMark, a novel framework designed to embed resilient watermarks into T2I diffusion models. SleeperMark explicitly guides the model to disentangle the watermark information from the semantic concepts it learns, allowing the model to retain the embedded watermark while continuing to be fine-tuned to new downstream tasks. Our extensive experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models, including latent diffusion models (e.g., Stable Diffusion) and pixel diffusion models (e.g., DeepFloyd-IF), showing robustness against downstream fine-tuning and various attacks at both the image and model levels, with minimal impact on the model's generative capability. The code is available at <a class="link-external link-https" href="https://github.com/taco-group/SleeperMark" rel="external noopener nofollow">this https URL</a>.

Countering Personalized Text-to-Image Generation with Influence Watermarks

Warfare:Breaking the Watermark Protection of AI-Generated Content

Generative Watermarking Against Unauthorized Subject-Driven Image Synthesis

FT-Shield: A Watermark Against Unauthorized Fine-tuning in Text-to-Image Diffusion Models

InvisMark: Invisible and Robust Watermarking for AI-generated Image Provenance

PersonaMark: Personalized LLM watermarking for model protection and user attribution

Watermark-embedded Adversarial Examples for Copyright Protection against Diffusion Models

Protect-Your-IP: Scalable Source-Tracing and Attribution Against Personalized Generation

Catch You Everything Everywhere: Guarding Textual Inversion via Concept Watermarking

Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage

Invisible Image Watermarks Are Provably Removable Using Generative AI

Steal My Artworks for Fine-tuning? A Watermarking Framework for Detecting Art Theft Mimicry in Text-to-Image Models

Watermarking Neural Networks with Watermarked Images

Invisible Watermarks: Attacks and Robustness

Robustness of Watermarking on Text-to-Image Diffusion Models

Protecting the Intellectual Properties of Deep Neural Networks with an Additional Class and Steganographic Images

Hide and Seek: How Does Watermarking Impact Face Recognition?

Go Wide or Go Deep: Levering Watermarking Performance with Computational Cost for Specific Images

On Evaluating The Performance of Watermarked Machine-Generated Texts Under Adversarial Attacks

SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models

Invisible Adversarial Watermarking: A Novel Security Mechanism for Enhancing Copyright Protection