Abstract:In this paper, we are concerned with differentially private stochastic gradient descent (SGD) algorithms in the setting of stochastic convex optimization (SCO). Most of the existing work requires the loss to be Lipschitz continuous and strongly smooth, and the model parameter to be uniformly bounded. However, these assumptions are restrictive as many popular losses violate these conditions including the hinge loss for SVM, the absolute loss in robust regression, and even the least square loss in an unbounded domain. We significantly relax these restrictive assumptions and establish privacy and generalization (utility) guarantees for private SGD algorithms using output and gradient perturbations associated with non-smooth convex losses. Specifically, the loss function is relaxed to have an α-Hölder continuous gradient (referred to as α-Hölder smoothness) which instantiates the Lipschitz continuity (α=0) and the strong smoothness (α=1). We prove that noisy SGD with α-Hölder smooth losses using gradient perturbation can guarantee (ϵ,δ)-differential privacy (DP) and attain optimal excess population risk O(dlog⁡(1/δ)nϵ+1n), up to logarithmic terms, with the gradient complexity O(n2−α1+α+n). This shows an important trade-off between α-Hölder smoothness of the loss and the computational complexity for private SGD with statistically optimal performance. In particular, our results indicate that α-Hölder smoothness with α≥1/2 is sufficient to guarantee (ϵ,δ)-DP of noisy SGD algorithms while achieving optimal excess risk with a linear gradient complexity O(n).

Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain

Differentially Private Regression with Unbounded Covariates

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

Differentially Private Simple Linear Regression

Differentially Private Domain Adaptation with Theoretical Guarantees

Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners

Better Private Linear Regression Through Better Private Feature Selection

Differentially private regression analysis with dynamic privacy allocation

Differentially Private Model Selection with Penalized and Constrained Likelihood

Differentially Private Generalized Linear Models Revisited

Differentially Private Learning Beyond the Classical Dimensionality Regime

Differentially Private SGD with Non-Smooth Losses

Private Estimation and Inference in High-Dimensional Regression with FDR Control

Differentially Private Linear Regression Analysis via Truncating Technique

Revisiting Differentially Private Hyper-parameter Tuning

Efficient Sparse Least Absolute Deviation Regression with Differential Privacy

Differentially Private ADMM Algorithms for Machine Learning

Differentially Private Sliced Inverse Regression: Minimax Optimality and Algorithm

Differential Privacy Preservation in Regression Analysis Based on Relevance

Differentially Private Log-Location-Scale Regression Using Functional Mechanism