Abstract:In order to rapidly train deep learning models, many adaptive gradient methods have been proposed in recent years, such as Adam and AMSGrad. However, the computation of the full gradient vectors in the above methods becomes expensive prohibitively at each iteration for handling high dimensional data. Moreover, the private information may be leaked in the process of training. For these reasons, we propose a differentially private randomized block-coordinate adaptive gradient algorithm, called DP-RBAdaBound, for training deep learning model. To reduce the computation of the full gradient vectors, we randomly choose a block-coordinate to update the model parameter at each iteration. Meanwhile, we add the Laplace noise to the block-coordinate of the gradient vector per iteration for preserving the privacy of users. Furthermore, we rigorously show that the proposed algorithm can preserve ϵ-differential privacy, where ϵ>0 denotes the privacy level. Moreover, we also rigorously prove that the square-root regret bound is also achieved in convex settings, i.e., O(T), where T is a time horizon. Besides, we offer a tradeoff between regret bound and privacy, i.e., the regret bound has order of O(1/ϵ4) by fixing other parameters when ϵ-differential privacy is achieved. Finally, we confirm the computational benefit by training DenseNet-121 and ResNet-34 models on CIFAR-10 dataset, respectively. Meanwhile, the effectiveness of DP-RBAdaBound is also validated through training the DenseNet-121 model on CIFAR-100 dataset and LSTM model on Penn TreeBank dataset, respectively.

DP-RBAdaBound: A Differentially Private Randomized Block-Coordinate Adaptive Gradient Algorithm for Training Deep Neural Networks

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Differentially Private Convolutional Neural Networks with Adaptive Gradient Descent.

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

A Stochastic Gradient Descent Algorithm Based on Adaptive Differential Privacy

DPAdaMod_AGC: Adaptive Gradient Clipping-Based Differential Privacy

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Dynamic Differential-Privacy Preserving SGD

Towards Efficient and Scalable Training of Differentially Private Deep Learning

GReDP: A More Robust Approach for Differential Private Training with Gradient-Preserving Noise Reduction

Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent

Improving Differentially Private SGD via Randomly Sparsified Gradients

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight

An Adaptive and Fast Convergent Approach to Differentially Private Deep Learning

Differentially Private Deep Learning With Dynamic Privacy Budget Allocation and Adaptive Optimization

Differentially Private Block-wise Gradient Shuffle for Deep Learning

DP-SGD with weight clipping

DP-FP: Differentially Private Forward Propagation for Large Models

Differentially Private Adaptive Optimization with Delayed Preconditioners