A Recipe for Watermarking Diffusion Models

Yunqing Zhao,Tianyu Pang,Chao Du,Xiao Yang,Ngai-Man Cheung,Min Lin

2023-10-15

Abstract:Diffusion models (DMs) have demonstrated advantageous potential on generative tasks. Widespread interest exists in incorporating DMs into downstream applications, such as producing or editing photorealistic images. However, practical deployment and unprecedented power of DMs raise legal issues, including copyright protection and monitoring of generated content. In this regard, watermarking has been a proven solution for copyright protection and content monitoring, but it is underexplored in the DMs literature. Specifically, DMs generate samples from longer tracks and may have newly designed multimodal structures, necessitating the modification of conventional watermarking pipelines. To this end, we conduct comprehensive analyses and derive a recipe for efficiently watermarking state-of-the-art DMs (e.g., Stable Diffusion), via training from scratch or finetuning. Our recipe is straightforward but involves empirically ablated implementation details, providing a foundation for future research on watermarking DMs. The code is available at <a class="link-external link-https" href="https://github.com/yunqing-me/WatermarkDM" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Cryptography and Security,Machine Learning

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily focuses on the issues of copyright protection and content monitoring in generative tasks using Diffusion Models (DMs). Specifically:

1. **Copyright Protection**: Pre-trained diffusion models (such as Stable Diffusion) are widely used in various practical applications, so it is necessary to ensure that these applications respect the copyright of the underlying pre-trained models and comply with relevant licenses. However, practical applications usually only provide black-box APIs, which do not allow direct inspection of the model's copyright or license.

2. **Detection of Generated Content**: Using generative models to create fake content (e.g., Deepfake), new artworks, or abusive materials may pose legal risks or disputes. As the capabilities of diffusion models enhance, detecting and monitoring these contents become more challenging.

To address these issues, the authors propose an effective method for embedding watermarks into state-of-the-art diffusion models, including unconditional/category-conditional generative models and text-to-image generative models. By training from scratch or fine-tuning, the authors provide a simple yet empirically analyzed implementation detail, laying the foundation for future research.