Abstract:Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on $(\varepsilon, \delta)$-DP ($\delta > 0$), whereas proving tight excess risk and runtime bounds for $(\varepsilon, 0)$-differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known $(\varepsilon, 0)$-differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Improve the gradient perturbation approach for differentially private optimization

Gradient Perturbation is Underrated for Differentially Private Convex Optimization

Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners

Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization

Normalized/clipped sgd with perturbation for differentially private non-convex optimization

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

Do not let privacy overbill utility: Gradient embedding perturbation for private learning

On the Differentially Private Nature of Perturbed Gradient Descent

Differentially Private Deep Learning with Iterative Gradient Descent Optimization

Do not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Tailoring Gradient Methods for Differentially-Private Distributed Optimization

Differentially Private Adaptive Optimization with Delayed Preconditioners

Output Perturbation for Differentially Private Convex Optimization: Faster and More General

Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budget

Beyond Uniform Lipschitz Condition in Differentially Private Optimization

Differentially Private Bilevel Optimization

Differentially private stochastic gradient descent with low-noise

Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent