Abstract:In this work, we conduct a systematic study of stochastic saddle point problems (SSP) and stochastic variational inequalities (SVI) under the constraint of $(\epsilon,\delta)$-differential privacy (DP) in both Euclidean and non-Euclidean setups. We first consider Lipschitz convex-concave SSPs in the $\ell_p/\ell_q$ setup, $p,q\in[1,2]$. Here, we obtain a bound of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$ on the strong SP-gap, where $n$ is the number of samples and $d$ is the dimension. This rate is nearly optimal for any $p,q\in[1,2]$. Without additional assumptions, such as smoothness or linearity requirements, prior work under DP has only obtained this rate when $p=q=2$ (i.e., only in the Euclidean setup). Further, existing algorithms have each only been shown to work for specific settings of $p$ and $q$ and under certain assumptions on the loss and the feasible set, whereas we provide a general algorithm for DP SSPs whenever $p,q\in[1,2]$. Our result is obtained via a novel analysis of the recursive regularization algorithm. In particular, we develop new tools for analyzing generalization, which may be of independent interest. Next, we turn our attention towards SVIs with a monotone, bounded and Lipschitz operator and consider $\ell_p$-setups, $p\in[1,2]$. Here, we provide the first analysis which obtains a bound on the strong VI-gap of $\tilde{O}\big(\frac{1}{\sqrt{n}} + \frac{\sqrt{d}}{n\epsilon}\big)$. For $p-1=\Omega(1)$, this rate is near optimal due to existing lower bounds. To obtain this result, we develop a modified version of recursive regularization. Our analysis builds on the techniques we develop for SSPs as well as employing additional novel components which handle difficulties arising from adapting the recursive regularization framework to SVIs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the optimization problems of stochastic saddle point problems (SSPs) and stochastic variational inequalities (SVIs) in Euclidean and non - Euclidean geometries under the constraint of differential privacy (DP). Specifically: 1. **Research Background**: - Stochastic saddle point problems (SSP) are an increasingly important part of the machine - learning toolkit. These problems have an inherent min - max structure, also known as stochastic min - max optimization problems. - Stochastic variational inequalities (SVI) are closely related to saddle point problems, but some problems that can be represented as monotone SVI are not easily transformed into convex - concave SSP. 2. **Problem Description**: - The author focuses on the systematic study of stochastic saddle point problems and stochastic variational inequalities under the $(\epsilon, \delta)$-differential privacy constraint, especially the problems in the $\ell_p / \ell_q$ setting. - The specific goal is to find an approximate solution such that the strong SP - gap (for SSP) or the strong VI - gap (for SVI) reaches the optimal or near - optimal convergence rate. 3. **Main Challenges**: - Under the constraint of differential privacy, previous work has mainly focused on Euclidean geometry, and the optimal convergence rate in non - Euclidean geometry has not been fully studied. - For SVI, even in Euclidean geometry, the best achievable utility is unknown. 4. **Solutions**: - The author proposes a new recursive regularization algorithm and develops new generalization analysis tools to deal with problems in non - Euclidean geometry. - For SSP, in the $\ell_p / \ell_q$ setting, they obtain an approximately optimal strong SP - gap convergence rate: \[ \tilde{O}\left(\frac{1}{\sqrt{n}}+\frac{\sqrt{d}}{n\epsilon}\right) \] - For SVI, in the $\ell_p$ setting, they also obtain a similar approximately optimal strong VI - gap convergence rate. 5. **Contributions**: - Provide the first work on the systematic study of stochastic saddle point problems and variational inequalities in Euclidean and non - Euclidean geometries. - Develop new analysis tools that are applicable not only to SSP but also to SVI and reach the optimal convergence rate in some cases. Through these methods, the author successfully extends the existing technology so that it can handle optimization problems under differential privacy constraints in a broader geometric context.

Private Algorithms for Stochastic Saddle Points and Variational Inequalities: Beyond Euclidean Geometry

Differentially Private Algorithms for the Stochastic Saddle Point Problem with Optimal Rates for the Strong Gap

Mirror Descent Algorithms with Nearly Dimension-Independent Rates for Differentially-Private Stochastic Saddle-Point Problems

General Procedure to Provide High-Probability Guarantees for Stochastic Saddle Point Problems

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited

Generalized Smooth Stochastic Variational Inequalities: Almost Sure Convergence and Convergence Rates

Private Stochastic Convex Optimization with Optimal Rates

User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Rates

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Beyond Uniform Lipschitz Condition in Differentially Private Optimization

Differentially Private Optimization with Sparse Gradients

Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates

Improved Sample Complexity for Private Nonsmooth Nonconvex Optimization

Faster Algorithms for User-Level Private Stochastic Convex Optimization

Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions

Differentially Private Distributed Stochastic Optimization with Time-Varying Sample Sizes

High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance

Private Robust Estimation by Stabilizing Convex Relaxations

Stochastic Methods in Variational Inequalities: Ergodicity, Bias and Refinements