Abstract:The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitably impairing the accuracy of the trained models. In this paper, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but teaches a student to be publicly released. In particular, a three-player (teacher-student-discriminator) learning framework is proposed to achieve trade-off between utility and privacy, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. We then integrate a differential privacy protection mechanism into the learning procedure, which enables a rigorous privacy budget for the training. The framework eventually allows student to be trained with only unlabelled public data and very few epochs, and hence prevents the exposure of sensitive training data, while ensuring model utility with a modest privacy budget. The experiments on MNIST, SVHN and CIFAR-10 datasets show that our students obtain the accuracy losses w.r.t teachers of 0.89%, 2.29%, 5.16%, respectively with the privacy bounds of (1.93, 10(-5)), (5.02, 10(-6)), (8.81, 10(-6)). When compared with the existing works [15, 20], the proposed work can achieve 582% accuracy loss improvement.

Can Differential Privacy Practically Protect Collaborative Deep Learning Inference for IoT?

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

A Distributed Privacy-Preserving Framework for Deep Learning with Edge-Cloud Computing.

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

Privacy-preserving Security Inference Towards Cloud-Edge Collaborative Using Differential Privacy

Over-the-Air Collaborative Inference with Feature Differential Privacy

RL-DistPrivacy: Privacy-Aware Distributed Deep Inference for low latency IoT systems

Performance-Enhanced Federated Learning with Differential Privacy for Internet of Things

PATROL: Privacy-Oriented Pruning for Collaborative Inference Against Model Inversion Attacks

Insuring against the perils in distributed learning: privacy-preserving empirical risk minimization

Practical Membership Inference Attack Against Collaborative Inference in Industrial IoT

Hybrid Differential Privacy Based Federated Learning for Internet of Things

Reconciling privacy and accuracy in AI for medical imaging

Aldp-fl: an adaptive local differential privacy-based federated learning mechanism for IoT

Deep Learning with Differential Privacy

Toward Training at ImageNet Scale with Differential Privacy

Gain Without Pain: Offsetting DP-Injected Noises Stealthily in Cross-Device Federated Learning

Protection Against Reconstruction and Its Applications in Private Federated Learning

Local Differential Privacy for Deep Learning

Collaborative Inference over Wireless Channels with Feature Differential Privacy

Enhancing IoT Security: A Novel Approach with Federated Learning and Differential Privacy Integration