Abstract:Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on $(\varepsilon, \delta)$-DP ($\delta > 0$), whereas proving tight excess risk and runtime bounds for $(\varepsilon, 0)$-differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known $(\varepsilon, 0)$-differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Differentially Private SGD with Non-Smooth Losses

Differentially private stochastic gradient descent with low-noise

Privacy-Preserving SGD on Shuffle Model

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization

Faster Algorithms for User-Level Private Stochastic Convex Optimization

Sharper Utility Bounds for Differentially Private Models: Smooth and Non-smooth

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Normalized/clipped sgd with perturbation for differentially private non-convex optimization

Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent

On the Privacy of Noisy Stochastic Gradient Descent for Convex Optimization

Shuffle Private Decentralized Convex Optimization

Output Perturbation for Differentially Private Convex Optimization: Faster and More General

Dynamic Differential-Privacy Preserving SGD

Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness

Private Stochastic Convex Optimization with Optimal Rates

DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM

Improving Differentially Private SGD via Randomly Sparsified Gradients

Differentially Private SGD with Random Features

Stochastic gradient descent with differentially private updates