Abstract:The integration of Differential Privacy (DP) with diffusion models (DMs) presents a promising yet challenging frontier, particularly due to the substantial memorization capabilities of DMs that pose significant privacy risks. Differential privacy offers a rigorous framework for safeguarding individual data points during model training, with Differential Privacy Stochastic Gradient Descent (DP-SGD) being a prominent implementation. Diffusion method decomposes image generation into iterative steps, theoretically aligning well with DP's incremental noise addition. Despite the natural fit, the unique architecture of DMs necessitates tailored approaches to effectively balance privacy-utility trade-off. Recent developments in this field have highlighted the potential for generating high-quality synthetic data by pre-training on public data (i.e., ImageNet) and fine-tuning on private data, however, there is a pronounced gap in research on optimizing the trade-offs involved in DP settings, particularly concerning parameter efficiency and model scalability. Our work addresses this by proposing a parameter-efficient fine-tuning strategy optimized for private diffusion models, which minimizes the number of trainable parameters to enhance the privacy-utility trade-off. We empirically demonstrate that our method achieves state-of-the-art performance in DP synthesis, significantly surpassing previous benchmarks on widely studied datasets (e.g., with only 0.47M trainable parameters, achieving a more than 35% improvement over the previous state-of-the-art with a small privacy budget on the CelebA-64 dataset). Anonymous codes available at https://anonymous.4open.science/r/DP-LORA-F02F.

On the Inherent Privacy Properties of Discrete Denoising Diffusion Models

Differentially Private Denoise Diffusion Probability Models.

Multi-Step Denoising Scheduled Sampling: Towards Alleviating Exposure Bias for Diffusion Models

Differentially Private Fine-Tuning of Diffusion Models

PAC Privacy Preserving Diffusion Models

Differentially Private Latent Diffusion Models

Leveraging Programmatically Generated Synthetic Data for Differentially Private Diffusion Training

Efficient Differentially Private Fine-Tuning of Diffusion Models

Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation

Data Forensics in Diffusion Models: A Systematic Analysis of Membership Privacy

Structured Denoising Diffusion Models in Discrete State-Spaces

Differentially Private Federated Learning of Diffusion Models for Synthetic Tabular Data Generation

Smoothed Differential Privacy

Gradient-Guided Conditional Diffusion Models for Private Image Reconstruction: Analyzing Adversarial Impacts of Differential Privacy and Denoising

Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings

Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation

Differentially-Private Deep Learning from an Optimization Perspective.

Denoising Lévy Probabilistic Models

PriSampler: Mitigating Property Inference of Diffusion Models

A Note on the Convergence of Denoising Diffusion Probabilistic Models