Abstract:In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

Discarding Lavrentiev's Gap in Non-automonous and Non-Convex Variational Problems

Avoidance of the Lavrentiev gap for one-dimensional non-autonomous functionals with constraints

Globally Lipschitz minimizers for variational problems with linear growth

The Lavrentiev phenomenon in calculus of variations with differential forms

Minimizers of L2-subcritical variational problems with spatially decaying nonlinearities in bounded domains

A Liouville Type Theorem for a Variational Problem with Free Boundary in Three Dimensions

The Lavrentiev phenomenon

A Duality Theory for Non-convex Problems in the Calculus of Variations

A priori $L^\infty-$bound for Ginzburg-Landau energy minimizers with divergence penalization

Variational competition between full Hessian and its determinant for convex functions

Uniqueness, regularity and characteristic flow for a non strictly convex singular variational problem

Uniqueness and characteristic flow for a non strictly convex singular variational problem

Free boundary regularity for semilinear variational problems with a topological constraint

Variational competition between the full Hessian and its determinant for convex functions

On quasi-inertial double extragradient-type iterations for variational inequalities and Bregman relatively asymptotically nonexpansive mapping

Algorithms for solving variational inequalities and saddle point problems with some generalizations of Lipschitz property for operators

Necessary and sufficient conditions for one-dimensional variational problems with applications to elasticity

Adaptive Methods for Variational Inequalities under Relaxed Smoothness Assumption

Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry