Privacy-Preserving Diffusion Model Using Homomorphic Encryption

Yaojian Chen,Qiben Yan

2024-03-09

Abstract:In this paper, we introduce a privacy-preserving stable diffusion framework leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses on protecting the denoising phase of the diffusion process. HE-Diffusion is a tailored encryption framework specifically designed to align with the unique architecture of stable diffusion, ensuring both privacy and functionality. To address the inherent computational challenges, we propose a novel min-distortion method that enables efficient partial image encryption, significantly reducing the overhead without compromising the model's output quality. Furthermore, we adopt a sparse tensor representation to expedite computational operations, enhancing the overall efficiency of the privacy-preserving diffusion process. We successfully implement HE-based privacy-preserving stable diffusion inference. The experimental results show that HE-Diffusion achieves 500 times speedup compared with the baseline method, and reduces time cost of the homomorphically encrypted inference to the minute level. Both the performance and accuracy of the HE-Diffusion are on par with the plaintext counterpart. Our approach marks a significant step towards integrating advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation in critical applications.

Cryptography and Security,Artificial Intelligence

What problem does this paper attempt to address?

The problem addressed in this paper is how to achieve efficient inference in a stable diffusion model while protecting user privacy. The paper "Privacy-preserving Diffusion with Homomorphic Encryption" presents a privacy-preserving diffusion framework called HE-Diffusion, which utilizes homomorphic encryption technology. This framework focuses on the denoising phase of the diffusion process, ensuring partial encryption of images without sacrificing output quality and reducing computational overhead. To address the computational challenges of homomorphic encryption, a novel minimal distortion method is proposed in the paper to achieve efficient partial image encryption. Additionally, the use of sparse tensor representation accelerates computational operations and improves the overall efficiency of the privacy-preserving diffusion process. Experimental results show that HE-Diffusion is 500 times faster than baseline methods, reducing the time cost of homomorphic encryption inference to the level of minutes. HE-Diffusion performs comparably to the plaintext version in terms of performance and accuracy. This approach represents an important step in combining advanced cryptographic techniques with state-of-the-art generative models, paving the way for privacy-preserving and efficient image generation, especially in sensitive applications such as healthcare, personal security, and confidential design.

Privacy-Preserving Diffusion Model Using Homomorphic Encryption

Differentially Private Denoise Diffusion Probability Models.

Diff-Privacy: Diffusion-based Face Privacy Protection

PAC Privacy Preserving Diffusion Models

Generation of Face Privacy-Protected Images Based on the Diffusion Model

Gradient-Guided Conditional Diffusion Models for Private Image Reconstruction: Analyzing Adversarial Impacts of Differential Privacy and Denoising

Data Forensics in Diffusion Models: A Systematic Analysis of Membership Privacy

Guided Diffusion Model for Sensor Data Obfuscation

TPE-H2MWD: an exact thumbnail preserving encryption scheme with hidden Markov model and weighted diffusion

Enabling Competitive Performance of Medical Imaging with Diffusion Model-generated Images without Privacy Leakage

On the Inherent Privacy Properties of Discrete Denoising Diffusion Models

Privacy-Preservation in Distributed Deep Neural Networks Via Encryption of Selected Gradients.

Secure Convolution Neural Network Inference Based on Homomorphic Encryption

Enhancing User-Centric Privacy Protection: An Interactive Framework through Diffusion Models and Machine Unlearning

Differentially Private Fine-Tuning of Diffusion Models

Image Encryption and Compression Based on Reversed Diffusion Model

Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation

Optimized Privacy-Preserving CNN Inference With Fully Homomorphic Encryption

Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings

Extracting Training Data from Diffusion Models

Privacy-preserving Decentralized Deep Learning with Multiparty Homomorphic Encryption