Abstract:Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on $(\varepsilon, \delta)$-DP ($\delta > 0$), whereas proving tight excess risk and runtime bounds for $(\varepsilon, 0)$-differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known $(\varepsilon, 0)$-differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Sharper Utility Bounds for Differentially Private Models: Smooth and Non-smooth

Differentially Private Generalized Linear Models Revisited

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Differentially private stochastic gradient descent with low-noise

Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners

Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization

Differentially private regression analysis with dynamic privacy allocation

Output Perturbation for Differentially Private Convex Optimization: Faster and More General

Differentially Private Variational Inference for Non-conjugate Models

Differentially Private Empirical Risk Minimization Revisited: Faster and More General

DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM

Certification for Differentially Private Prediction in Gradient-Based Training

Differentially Private Worst-group Risk Minimization

Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics

Do not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning

Faster Rates of Private Stochastic Convex Optimization

The Cost of Privacy in Generalized Linear Models: Algorithms and Minimax Lower Bounds

Beyond Uniform Lipschitz Condition in Differentially Private Optimization

Optimal Differentially Private Model Training with Public Data

Privacy for Free in the Over-Parameterized Regime

Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates