Abstract:The deployment of deep learning applications has to address the increasing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information from either model parameters or accesses to the inference model. Recently, differential privacy (DP) has been proposed to offer provable privacy guarantees by randomizing the training process of neural networks. However, many approaches tend to provide the worst case privacy protection for model publishing, inevitably impairing the accuracy of the trained models. Thus, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but the student models can be released with privacy guarantees. In this paper, a three-player (teacher-student-discriminator) learning framework, Private Knowledge Distillation with Generative Adversarial Networks (PKDGAN), is proposed, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. Moreover, a cooperative learning strategy is also suggested to support the collective training of multiple students against the discriminator when each student is with insufficient unlabelled training data. To enforce rigorous privacy guarantees, PKDGAN applies a Rényi differential privacy mechanism throughout the training process, and use it with a moment accountant technique to track the privacy cost. PKDGAN allows students to be trained with unlabelled public data and very few epochs, which avoids the exposure of training data while ensuring model performance. In the experiments, PKDGAN is found to have consistently good performance on various datasets (MNIST, SVHN, CIFAR-10, and Market-1501). When compared to prior works [1], [2], PKDGAN exhibits 5-82% accuracy loss improvement without compromising any privacy guarantee.

PPNNI: Privacy-Preserving Neural Network Inference Against Adversarial Example Attack

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

PKDGAN: Private Knowledge Distillation with Generative Adversarial Networks

Industrial Scale Privacy Preserving Deep Neural Network

Privacy-Enhanced Federated GNN Inference Against Adversarial Example Attack

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

Achieving Efficient and Privacy-Preserving Neural Network Training and Prediction in Cloud Environments

Secure Neural Network Prediction in the Cloud-Based Open Neural Network Service

Practical Two-party Privacy-preserving Neural Network Based on Secret Sharing

Privacy-Preserving DNN Model Authorization Against Model Theft and Feature Leakage

Only My Model On My Data: A Privacy Preserving Approach Protecting one Model and Deceiving Unauthorized Black-Box Models

PrivyNet: A Flexible Framework for Privacy-Preserving Deep Neural Network Training

C2PI: An Efficient Crypto-Clear Two-Party Neural Network Private Inference

Secure Prediction of Neural Network in the Cloud

Optimizing Privacy-Preserving Outsourced Convolutional Neural Network Predictions

Invisible Adversarial Attack Against Deep Neural Networks: an Adaptive Penalization Approach

Privacy-Preserving Adversarial Networks

Crypto-Nets: Neural Networks over Encrypted Data

Gain Without Pain: Offsetting DP-Injected Noises Stealthily in Cross-Device Federated Learning

Enabling Inference Privacy with Adaptive Noise Injection

PriPHiT: Privacy-Preserving Hierarchical Training of Deep Neural Networks