Abstract:The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitably impairing the accuracy of the trained models. In this paper, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but teaches a student to be publicly released. In particular, a three-player (teacher-student-discriminator) learning framework is proposed to achieve trade-off between utility and privacy, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. We then integrate a differential privacy protection mechanism into the learning procedure, which enables a rigorous privacy budget for the training. The framework eventually allows student to be trained with only unlabelled public data and very few epochs, and hence prevents the exposure of sensitive training data, while ensuring model utility with a modest privacy budget. The experiments on MNIST, SVHN and CIFAR-10 datasets show that our students obtain the accuracy losses w.r.t teachers of 0.89%, 2.29%, 5.16%, respectively with the privacy bounds of (1.93, 10(-5)), (5.02, 10(-6)), (8.81, 10(-6)). When compared with the existing works [15, 20], the proposed work can achieve 582% accuracy loss improvement.

Differentially Private Gradient Flow based on the Sliced Wasserstein Distance

Private Gradient Estimation is Useful for Generative Modeling

Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

Differentially Private Sliced Wasserstein Distance

Differentially Private Synthetic Data Generation via Lipschitz-Regularised Variational Autoencoders

Differentially Private Variational Inference for Non-conjugate Models

Model Conversion via Differentially Private Data-Free Distillation

Stochastic gradient descent with differentially private updates

FedGG: Leveraging Generative Adversarial Networks and Gradient Smoothing for Privacy Protection in Federated Learning.

Are Normalizing Flows the Key to Unlocking the Exponential Mechanism? A Path through the Accuracy-Privacy Ceiling Constraining Differentially Private ML

Differentially Private Generative Adversarial Network

Do not let privacy overbill utility: Gradient embedding perturbation for private learning

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

Scalable Differentially Private Generative Student Model via PATE

Differentially Private SGD with Random Features

Differentially Private Learning with Grouped Gradient Clipping.

Training generative models from privatized data

Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation

Gaussian-Smoothed Sliced Probability Divergences

Federated Synthetic Data Generation with Differential Privacy