Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking

Zhiyuan Ma,Guoli Jia,Biqing Qi,Bowen Zhou

2024-07-19

Abstract:Recently, stable diffusion (SD) models have typically flourished in the field of image synthesis and personalized editing, with a range of photorealistic and unprecedented images being successfully generated. As a result, widespread interest has been ignited to develop and use various SD-based tools for visual content creation. However, the exposure of AI-created content on public platforms could raise both legal and ethical risks. In this regard, the traditional methods of adding watermarks to the already generated images (i.e. post-processing) may face a dilemma (e.g., being erased or modified) in terms of copyright protection and content monitoring, since the powerful image inversion and text-to-image editing techniques have been widely explored in SD-based methods. In this work, we propose a Safe and high-traceable Stable Diffusion framework (namely Safe-SD) to adaptively implant the graphical watermarks (e.g., QR code) into the imperceptible structure-related pixels during the generative diffusion process for supporting text-driven invisible watermarking and detection. Different from the previous high-cost injection-then-detection training framework, we design a simple and unified architecture, which makes it possible to simultaneously train watermark injection and detection in a single network, greatly improving the efficiency and convenience of use. Moreover, to further support text-driven generative watermarking and deeply explore its robustness and high-traceability, we elaborately design lambda sampling and encryption algorithm to fine-tune a latent diffuser wrapped by a VAE for balancing high-fidelity image synthesis and high-traceable watermark detection. We present our quantitative and qualitative results on two representative datasets LSUN, COCO and FFHQ, demonstrating state-of-the-art performance of Safe-SD and showing it significantly outperforms the previous approaches.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the legal and ethical risks posed by powerful generative models (such as Stable Diffusion) in image synthesis and personalized editing. Specifically, the researchers propose a new framework called Safe-SD, which adaptively embeds graphical watermarks (e.g., QR codes) during the generative diffusion process, thereby supporting text-driven invisible watermark injection and detection. Compared to traditional methods that add watermarks after image generation, Safe-SD can embed watermarks directly during the generation process, avoiding the risk of watermarks being easily erased or modified.

The key contributions of the paper include:
1. **Unified Watermark Injection and Tracking Framework**: Safe-SD designs a framework that can simultaneously train watermark injection and detection in a single network, balancing the needs for high-fidelity image synthesis and highly traceable watermark detection, thereby improving training efficiency and ease of use.
2. **Invisible Structure-Related Pixel Embedding**: It can embed graphical watermarks into invisible structure-related pixels, ensuring that they are difficult to erase or modify even when facing image inversion or editing models.
3. **Support for Multiple Application Scenarios**: Suitable for a wide range of tasks, such as text-to-image synthesis, text-based image editing, and multi-watermark injection.
4. **Enhanced Robustness and Security**: Introduces fine-tuned latent diffusers and a carefully designed λ encryption algorithm, enhancing defense against attacks, and conducts detailed experimental validation.

Experimental results on three representative datasets, LSUN, COCO, and FFHQ, show that Safe-SD achieves the current best level in invisible watermark technology and significantly outperforms previous methods.