Abstract:In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

Taming under isoperimetry

Tamed Langevin sampling under weaker conditions

Non-asymptotic convergence bounds for modified tamed unadjusted Langevin algorithm in non-convex setting

Proximal Langevin Algorithm: Rapid Convergence Under Isoperimetry

Fast parallel sampling under isoperimetry

Taming the Interacting Particle Langevin Algorithm -- the superlinear case

Non-asymptotic convergence bounds for Sinkhorn iterates and their gradients: a coupling approach

Concentration of the Langevin Algorithm's Stationary Distribution

Convergence Of The Unadjusted Langevin Algorithm For Discontinuous Gradients

Analysis of Primal-Dual Langevin Algorithms

Primal Dual Interpretation of the Proximal Stochastic Gradient Langevin Algorithm

Convergence of Unadjusted Langevin in High Dimensions: Delocalization of Bias

Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry

Projected Langevin Monte Carlo algorithms in non-convex and super-linear setting

Rate of convergence of the smoothed empirical Wasserstein distance

Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem

Linear convergence of proximal descent schemes on the Wasserstein space

Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity - the Strongly Convex Case

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

Projected Langevin dynamics and a gradient flow for entropic optimal transport

Subgradient Langevin Methods for Sampling from Non-smooth Potentials