Abstract:Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private stochastic convex optimization (SCO), which corresponds to population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM/SCO setups. Further, most recent work in private SCO has focused on $(\varepsilon, \delta)$-DP ($\delta > 0$), whereas proving tight excess risk and runtime bounds for $(\varepsilon, 0)$-differential privacy remains a challenging open problem. Our first contribution is to provide the tightest known $(\varepsilon, 0)$-differentially private expected population loss bounds and fastest runtimes for smooth and strongly convex loss functions. In particular, for SCO with well-conditioned smooth and strongly convex loss functions, we provide a linear-time algorithm with optimal excess risk. For our second contribution, we study DP optimization for a broad class of tilted loss functions-which can be used to promote fairness or robustness, and are not necessarily of ERM form. We establish the first known DP excess risk and runtime bounds for optimizing this class; under smoothness and strong convexity assumptions, our bounds are near optimal. For our third contribution, we specialize our theory to DP adversarial training. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

Differentially Private Range Queries with Correlated Input Perturbation

Differentially Private SQL with Bounded User Contribution

Perturb-and-Project: Differentially Private Similarities and Marginals

The Optimization Of The Range-Count Queries In Differential Privacy

Optimum Noise Mechanism for Differentially Private Queries in Discrete Finite Sets

Private Query Release via the Johnson-Lindenstrauss Transform

Gradient Perturbation is Underrated for Differentially Private Convex Optimization

A Range Query Scheme for Spatial Data with Shuffled Differential Privacy

Output Perturbation for Differentially Private Convex Optimization: Faster and More General

On the Differentially Private Nature of Perturbed Gradient Descent

Differentially Private Distributed Stochastic Optimization with Time-Varying Sample Sizes

Differentially private publication for related POI discovery

Differentially Private Distributed Convex Optimization via Functional Perturbation

Relaxed Marginal Consistency for Differentially Private Query Answering

Optimal error of query sets under the differentially-private matrix mechanism

SMT-based Query Tracking for Differentially Private Data Analytics Systems

Data Level Privacy Preserving: A Stochastic Perturbation Approach based on Differential Privacy

Distributed Differentially Private Computation of Functions with Correlated Noise

Exact Inference with Approximate Computation for Differentially Private Data via Perturbations

A Differential Privacy Mechanism Design Under Matrix-Valued Query