Abstract:The deployment of deep learning applications has to address the increasing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information from either model parameters or accesses to the inference model. Recently, differential privacy (DP) has been proposed to offer provable privacy guarantees by randomizing the training process of neural networks. However, many approaches tend to provide the worst case privacy protection for model publishing, inevitably impairing the accuracy of the trained models. Thus, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but the student models can be released with privacy guarantees. In this paper, a three-player (teacher-student-discriminator) learning framework, Private Knowledge Distillation with Generative Adversarial Networks (PKDGAN), is proposed, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. Moreover, a cooperative learning strategy is also suggested to support the collective training of multiple students against the discriminator when each student is with insufficient unlabelled training data. To enforce rigorous privacy guarantees, PKDGAN applies a Rényi differential privacy mechanism throughout the training process, and use it with a moment accountant technique to track the privacy cost. PKDGAN allows students to be trained with unlabelled public data and very few epochs, which avoids the exposure of training data while ensuring model performance. In the experiments, PKDGAN is found to have consistently good performance on various datasets (MNIST, SVHN, CIFAR-10, and Market-1501). When compared to prior works [1], [2], PKDGAN exhibits 5-82% accuracy loss improvement without compromising any privacy guarantee.

A PATE-based Approach for Training Graph Neural Networks under Label Differential Privacy

PKDGAN: Private Knowledge Distillation with Generative Adversarial Networks

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

Achieving Adaptive Privacy-Preserving Graph Neural Networks Training in Cloud Environment

Devil in Disguise: Breaching Graph Neural Networks Privacy through Infiltration

Releasing Graph Neural Networks with Differential Privacy Guarantees

P$$^2$$CG: a Privacy Preserving Collaborative Graph Neural Network Training Framework

GraphPub: Generation of Differential Privacy Graph with High Availability

Local Differential Privacy in Graph Neural Networks: a Reconstruction Approach

Towards Robust Graph Neural Networks against Label Noise

Privacy-Enhanced Graph Neural Network for Decentralized Local Graphs

Efficient Privacy Preserving Graph Neural Network for Node Classification

Differentially Private Graph Neural Networks for Graph Classification and Its Adaptive Optimization

Edge Private Graph Neural Networks with Singular Value Perturbation

LPGNet: Link Private Graph Networks for Node Classification

Calibrating Privacy Budgets for Locally Private Graph Neural Networks

GReDP: A More Robust Approach for Differential Private Training with Gradient-Preserving Noise Reduction

Differentially Private Generative Adversarial Network

Adaptive Differential Privacy Mechanism Based on Entropy Theory for Preserving Deep Neural Networks

Towards Private Learning on Decentralized Graphs With Local Differential Privacy

Private-kNN: Practical Differential Privacy for Computer Vision