Abstract:We study the problem of differentially-private (DP) stochastic (convex-concave) saddle-points in the polyhedral setting. We propose $(\varepsilon, \delta)$-DP algorithms based on stochastic mirror descent that attain nearly dimension-independent convergence rates for the expected duality gap, a type of guarantee that was known before only for bilinear objectives. For convex-concave and first-order-smooth stochastic objectives, our algorithms attain a rate of $\sqrt{\log(d)/n} + (\log(d)^{3/2}/[n\varepsilon])^{1/3}$, where $d$ is the dimension of the problem and $n$ the dataset size. Under an additional second-order-smoothness assumption, we improve the rate on the expected gap to $\sqrt{\log(d)/n} + (\log(d)^{3/2}/[n\varepsilon])^{2/5}$. Under this additional assumption, we also show, by using bias-reduced gradient estimators, that the duality gap is bounded by $\log(d)/\sqrt{n} + \log(d)/[n\varepsilon]^{1/2}$ with constant success probability. This result provides evidence of the near-optimality of the approach. Finally, we show that combining our methods with acceleration techniques from online learning leads to the first algorithm for DP Stochastic Convex Optimization in the polyhedral setting that is not based on Frank-Wolfe methods. For convex and first-order-smooth stochastic objectives, our algorithms attain an excess risk of $\sqrt{\log(d)/n} + \log(d)^{7/10}/[n\varepsilon]^{2/5}$, and when additionally assuming second-order-smoothness, we improve the rate to $\sqrt{\log(d)/n} + \log(d)/\sqrt{n\varepsilon}$. Instrumental to all of these results are various extensions of the classical Maurey Sparsification Lemma, which may be of independent interest.

Decentralized saddle point problems via non-Euclidean mirror prox

A Decentralized Proximal Point-type Method for Non-convex Non-concave Saddle Point Problems

A Decentralized Proximal Point-type Method for Saddle Point Problems

Decentralized Distributed Optimization for Saddle Point Problems

The Mirror-Prox Sliding Method for Non-smooth decentralized saddle-point problems

Stochastic Mirror Descent for Convex Optimization with Consensus Constraints

On Decentralized Nonsmooth Optimization

Decentralized Proximal Optimization Method with Consensus Procedure

Switch and Conquer: Efficient Algorithms By Switching Stochastic Gradient Oracles For Decentralized Saddle Point Problems

Decentralized Optimization with Coupled Constraints

Mirror Descent Algorithms with Nearly Dimension-Independent Rates for Differentially-Private Stochastic Saddle-Point Problems

A Unified Distributed Method for Constrained Networked Optimization via Saddle-Point Dynamics

Distributed Saddle-Point Problems Under Similarity

Distributed Coordination for Nonsmooth Convex Optimization via Saddle-Point Dynamics

Decentralized Convex Optimization on Time-Varying Networks with Application to Wasserstein Barycenters

A Proximal Gradient Algorithm for Decentralized Nondifferentiable Optimization.

On Nonconvex Decentralized Gradient Descent

Convergence Analysis of Stochastic Saddle Point Mirror Descent Algorithm -- A Projected Dynamical View Point

Primal-dual Accelerated Mirror-Descent Method for Constrained Bilinear Saddle-Point Problems

Decentralized Optimization with Distributed Features and Non-Smooth Objective Functions

Distributed Multiproximal Algorithm for Nonsmooth Convex Optimization With Coupled Inequality Constraints