Abstract:The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitably impairing the accuracy of the trained models. In this paper, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but teaches a student to be publicly released. In particular, a three-player (teacher-student-discriminator) learning framework is proposed to achieve trade-off between utility and privacy, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. We then integrate a differential privacy protection mechanism into the learning procedure, which enables a rigorous privacy budget for the training. The framework eventually allows student to be trained with only unlabelled public data and very few epochs, and hence prevents the exposure of sensitive training data, while ensuring model utility with a modest privacy budget. The experiments on MNIST, SVHN and CIFAR-10 datasets show that our students obtain the accuracy losses w.r.t teachers of 0.89%, 2.29%, 5.16%, respectively with the privacy bounds of (1.93, 10(-5)), (5.02, 10(-6)), (8.81, 10(-6)). When compared with the existing works [15, 20], the proposed work can achieve 582% accuracy loss improvement.

Privacy-Preserving Split Learning Via Pareto Optimal Search.

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

Evaluating Privacy Leakage in Split Learning

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

Protecting Split Learning by Potential Energy Loss

Make Split, not Hijack: Preventing Feature-Space Hijacking Attacks in Split Learning

Secure Split Learning Against Property Inference, Data Reconstruction, and Feature Space Hijacking Attacks

Split Learning without Local Weight Sharing to Enhance Client-side Data Privacy

Split Ways: Privacy-Preserving Training of Encrypted Data Using Split Learning

PPCL: Privacy-preserving collaborative learning for mitigating indirect information leakage

Enhancing Accuracy-Privacy Trade-off in Differentially Private Split Learning

Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix

Feasibility Study of Multi-Site Split Learning for Privacy-Preserving Medical Systems under Data Imbalance Constraints in COVID-19, X-Ray, and Cholesterol Dataset

PrivyNet: A Flexible Framework for Privacy-Preserving Deep Neural Network Training

Towards Secure and Practical Machine Learning Via Secret Sharing and Random Permutation

Split-and-Denoise: Protect large language model inference with local differential privacy

Spatio-Temporal Split Learning for Privacy-Preserving Medical Platforms: Case Studies with COVID-19 CT, X-Ray, and Cholesterol Data

PPSFL: Privacy-Preserving Split Federated Learning for heterogeneous data in edge-based Internet of Things

SecureML: A System for Scalable Privacy-Preserving Machine Learning

Privacy-preserving two-parties logistic regression on vertically partitioned data using asynchronous gradient sharing

Insuring against the perils in distributed learning: privacy-preserving empirical risk minimization