Abstract:We introduce novel methods for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational overhead and degrade model performance when applied to large, complex models. Our approach instead leverages embedding-based techniques: Universal Guidance and Textual Inversion (TI), adapted with differentially private mechanisms. We apply these methods to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees, while both methods maintain high privacy resilience by employing calibrated noise and subsampling strategies. Our findings demonstrate a feasible and efficient pathway for privacy-preserving diffusion model adaptation, balancing data protection with the fidelity of generated images, and offer insights into embedding-driven methods for DP in generative AI applications.
Computer Vision and Pattern Recognition,Cryptography and Security,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to achieve privacy - protected adaptation of diffusion models on small - scale sensitive data sets while maintaining high fidelity. Specifically, traditional methods such as DP - SGD (Differentially Private Stochastic Gradient Descent) can provide privacy protection, but when applied to large and complex models, they will introduce significant computational overhead and lead to a decline in model performance. Therefore, this paper proposes a new method based on embedding technology. By combining Universal Guidance and Textual Inversion (TI) with the differential privacy mechanism, it can achieve privacy - protected style and content transfer without fine - tuning the model.
### Main Problems and Challenges
1. **Risk of Privacy Attacks**: Diffusion models are vulnerable to privacy attacks, such as membership inference attacks and data extraction attacks. These problems are particularly serious during the fine - tuning stage because at this time, usually smaller specialized data sets are used, and each data record has a greater impact on the final model.
2. **Limitations of DP - SGD**: Although DP - SGD can be theoretically applied to train diffusion models, there are many challenges in its practical application, such as large computational and memory overhead, incompatibility with batch processing operations, and a significant decline in performance when training on larger networks.
3. **Deficiencies of Existing Methods**: Existing privacy - protection methods (such as DP - LoRA, DP - RDM, etc.) are effective, but they still face certain limitations on small - scale data sets, especially when it is necessary to directly update model parameters.
### Proposed Solutions
The paper proposes two new privacy - protected adaptation methods:
1. **Privacy - Protected Style Guidance Based on Universal Guidance**: By using differentially private CLIP embeddings and calculating the embedding center points of multiple target images, and adding calibration noise to ensure differential privacy. This method can achieve style transfer without modifying the diffusion model itself.
2. **Privacy - Protected Text Inversion Based on Textual Inversion**: Learn an independent embedding vector for each target image, and then aggregate these embedding vectors into a noisy center point, thereby ensuring differential privacy while retaining the data compression advantages of TI.
### Experimental Results
The experimental results show that the method based on Textual Inversion can better maintain the fidelity of style transfer under privacy protection, and can generate high - quality images even under strict privacy guarantees. In addition, the subsampling strategy has been proven to be an effective means to improve privacy robustness, which can significantly reduce the impact of noise on image quality while controlling the sensitivity of individual data points.
### Conclusions
The methods proposed in this paper provide a feasible and efficient path for the adaptation of diffusion models under privacy protection, which can maintain the quality of generated images while protecting data privacy. This provides an important reference for the future application of diffusion models in privacy - sensitive fields (such as medical imaging, personal art works, etc.).
### Summary of Key Formulas
- **Differential Privacy Embedding Vector**:
\[
u^*=\frac{1}{n}\sum_{i = 1}^{n}u(i)+N(0,\sigma^2I)
\]
where the minimum value of \(\sigma\) is given by the following expression:
\[
\sigma=\frac{\Delta}{n}\sqrt{2\log(1.25 / \delta)} / \epsilon
\]
Here \(\Delta=\sup_{i,j}\|u(i)-u(j)\|\), and for normalized direction embedding vectors, \(\Delta = 2\).
- **Subsampled Embedding Vector**:
\[
u^*=\frac{1}{m}\sum_{u(i)\in D_{\text{sub}}}u(i)+N(0,\sigma^2I)
\]
where \(D_{\text{sub}}\subseteq\{u(1),\cdots\)