Abstract:The deployment of deep learning applications has to address the growing privacy concerns when using private and sensitive data for training. A conventional deep learning model is prone to privacy attacks that can recover the sensitive information of individuals from either model parameters or accesses to the target model. Recently, differential privacy that offers provable privacy guarantees has been proposed to train neural networks in a privacy-preserving manner to protect training data. However, many approaches tend to provide the worst case privacy guarantees for model publishing, inevitably impairing the accuracy of the trained models. In this paper, we present a novel private knowledge transfer strategy, where the private teacher trained on sensitive data is not publicly accessible but teaches a student to be publicly released. In particular, a three-player (teacher-student-discriminator) learning framework is proposed to achieve trade-off between utility and privacy, where the student acquires the distilled knowledge from the teacher and is trained with the discriminator to generate similar outputs as the teacher. We then integrate a differential privacy protection mechanism into the learning procedure, which enables a rigorous privacy budget for the training. The framework eventually allows student to be trained with only unlabelled public data and very few epochs, and hence prevents the exposure of sensitive training data, while ensuring model utility with a modest privacy budget. The experiments on MNIST, SVHN and CIFAR-10 datasets show that our students obtain the accuracy losses w.r.t teachers of 0.89%, 2.29%, 5.16%, respectively with the privacy bounds of (1.93, 10(-5)), (5.02, 10(-6)), (8.81, 10(-6)). When compared with the existing works [15, 20], the proposed work can achieve 582% accuracy loss improvement.

Iterative Privileged Learning

A Hybrid-Domain Framework for Secure Gradient Tree Boosting.

Coarse-grained privileged learning for classification

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

Learning with Privileged Information Using Bayesian Networks

Retaining Privileged Information for Multi-Task Learning

Learning Using Multiple-Type Privileged Information and SVM+ThinkTank.

Learning with Privileged Information for Multi-Label Classification

Learning Using Generated Privileged Information by Text-to-Image Diffusion Models

S-BDT: Distributed Differentially Private Boosted Decision Trees

Gradient Boosting With Piece-Wise Linear Regression Trees

An Interactive Machine Learning Framework

Privileged Label Enhancement with Multi-Label Learning.

Privileged Multi-label Learning

LSTSVR-PI: Least square twin support vector regression with privileged information

Boosting Alzheimer Diagnosis Accuracy with the Help of Incomplete Privileged Information

SecureBoost+: Large Scale and High-Performance Vertical Federated Gradient Boosting Decision Tree

FDPBoost: Federated differential privacy gradient boosting decision trees

Privet: A Privacy-Preserving Vertical Federated Learning Service for Gradient Boosted Decision Tables

Adaptive SVM+: Learning with Privileged Information for Domain Adaptation

Rethinking Knowledge Transfer in Learning Using Privileged Information