Abstract:Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on $(\varepsilon, \delta)$-DP ($\delta > 0$), whereas proving tight excess risk and runtime bounds for $(\varepsilon, 0)$-differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known $(\varepsilon, 0)$-differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Efficient private SCO for heavy-tailed data via averaged clipping

Efficient Private SCO for Heavy-Tailed Data via Averaged Clipping

Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates

Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions

Characterizing Private Clipped Gradient Descent on Convex Generalized Linear Problems.

Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails

Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization

Beyond Uniform Lipschitz Condition in Differentially Private Optimization

Private Stochastic Convex Optimization with Optimal Rates

High Probability Analysis for Non-Convex Stochastic Optimization with Clipping

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Smoothed Gradient Clipping and Error Feedback for Decentralized Optimization under Symmetric Heavy-Tailed Noise

Differentially Private Distributed Resource Allocation Via Deviation Tracking

Faster Rates of Private Stochastic Convex Optimization

Faster Algorithms for User-Level Private Stochastic Convex Optimization

DP-SGD with weight clipping

Differentially Private Learning with Per-Sample Adaptive Clipping.

Improving Differentially Private SGD via Randomly Sparsified Gradients

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited

Output Perturbation for Differentially Private Convex Optimization: Faster and More General