Abstract:DP-SGD has emerged as a popular method to protect personally identifiable information in deep learning applications. Unfortunately, DP-SGD's per-sample gradient clipping and uniform noise addition during training can significantly degrade model utility. To enhance the model's utility, researchers proposed various adaptive DP-SGD methods. However, we examine and discover that these techniques result in greater privacy leakage or lower accuracy than the traditional DP-SGD method, or a lack of evaluation on a complex data set such as CIFAR100. To address these limitations, we propose an Auto DP-SGD. Our method automates clipping threshold estimation based on the DL model's gradient norm and scales the gradients of each training sample without losing gradient information. This helps to improve the algorithm's utility while using a less privacy budget. To further improve accuracy, we introduce automatic noise multiplier decay mechanisms to decrease the noise multiplier after every epoch. Finally, we develop closed-form mathematical expressions using tCDP accountant for automatic noise multiplier and automatic clipping threshold estimation. Through extensive experimentation, we demonstrate that Auto DP-SGD outperforms existing SOTA DP-SGD methods in privacy and accuracy on various benchmark datasets. We also show that privacy can be improved by lowering the scale factor and using learning rate schedulers without significantly reducing accuracy. Specifically, Auto DP-SGD, when used with a step noise multiplier, improves accuracy by 3.20, 1.57, 6.73, and 1.42 for the MNIST, CIFAR10, CIFAR100, and AG News Corpus datasets, respectively. Furthermore, it obtains a substantial reduction in the privacy budget of 94.9, 79.16, 67.36, and 53.37 for the corresponding data sets.

DPAdaMod_AGC: Adaptive Gradient Clipping-Based Differential Privacy

DP-SGD with weight clipping

Dynamic Differential-Privacy Preserving SGD

A Stochastic Gradient Descent Algorithm Based on Adaptive Differential Privacy

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight

Differential Privacy Protection over Deep Learning: an Investigation of Its Impacted Factors

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Differentially Private Learning with Per-Sample Adaptive Clipping.

Differentially Private Convolutional Neural Networks with Adaptive Gradient Descent.

DP-RBAdaBound: A Differentially Private Randomized Block-Coordinate Adaptive Gradient Algorithm for Training Deep Neural Networks

AdaDP-CFL: Cluster Federated Learning with Adaptive Clipping Threshold Differential Privacy

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

AWDP-FL: An Adaptive Differential Privacy Federated Learning Framework

Auto DP-SGD: Dual Improvements of Privacy and Accuracy via Automatic Clipping Threshold and Noise Multiplier Estimation

Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classification

Differentially Private Learning with Grouped Gradient Clipping.

An Improved Stochastic Gradient Descent Algorithm Based on Rényi Differential Privacy

GReDP: A More Robust Approach for Differential Private Training with Gradient-Preserving Noise Reduction