Abstract:We propose a new stochastic method SAPD+ for solving nonconvex-concave minimax problems of the form $\min\max\mathcal{L}(x,y)=f(x)+\Phi(x,y)-g(y)$, where $f,g$ are closed convex and $\Phi(x,y)$ is a smooth function that is weakly convex in $x$, (strongly) concave in $y$. Let $\delta^2$ denote the variance bound for the unbiased stochastic oracle used within SAPD+ to estimate $\nabla\Phi$. When $\delta>0$, for both strongly concave and merely concave settings, SAPD+ achieves the best known oracle complexities: $\mathcal{O}\Big(\kappa_y\max\Big\{1,\frac{\delta^2}{\epsilon^2}\Big\}\frac{L\mathcal{G}_0}{\epsilon^{2}}\Big)$ for the strongly concave case without assuming compactness of the problem domain, and $\mathcal{O}\Big(\frac{L^3\mathcal{D}_y^2\mathcal{G}_0}{\epsilon^{4}}\Big(1+\frac{\delta^2}{\epsilon^2}\Big)\Big)$ for the merely concave case, where $\kappa_y\geq 1$ is the condition number, $L$ is the Lipschitz constant of $\nabla \Phi$, $\mathcal{G}_0$ is the primal-dual gap of the initial point, and $\mathcal{D}_y=\sup\{\|y\|:\ y\in\mathbf{dom} g\}$. We also propose SAPD+ with variance reduction, which enjoys $\mathcal{O}\Big(\max\Big\{\kappa_y,\sqrt{\frac{\delta}{\epsilon}}\Big\}\cdot (1+\kappa_y\frac{\delta}{\epsilon})\frac{L\mathcal{G}_0}{\epsilon^2}\Big)$ oracle complexity for weakly convex-strongly concave setting --this is the best known upper complexity bound in the literature for this setting and our paper establishes it for the first time. We demonstrate the efficiency of SAPD+ on a distributionally robust learning problem with a nonconvex regularizer and also on a multi-class classification problem in deep learning.

Robust Accelerated Primal-Dual Methods for Computing Saddle Points

Robust Accelerated Primal-Dual Methods for Computing Saddle Points

High Probability and Risk-Averse Guarantees for a Stochastic Accelerated Primal-Dual Method

Optimal Primal-Dual Methods for a Class of Saddle Point Problems

Accelerated Primal-dual Scheme for a Class of Stochastic Nonconvex-concave Saddle Point Problems

Randomized First-Order Methods for Saddle Point Optimization

SAPD+: An Accelerated Stochastic Method for Nonconvex-Concave Minimax Problems

Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling

A Randomized Block-Coordinate Primal-Dual Method for Large-scale Stochastic Saddle Point Problems

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Stochastic Parallel Block Coordinate Descent for Large-scale Saddle Point Problems

Switch and Conquer: Efficient Algorithms By Switching Stochastic Gradient Oracles For Decentralized Saddle Point Problems

A Stochastic Proximal Point Algorithm for Saddle-Point Problems

An efficient symmetric primal-dual algorithmic framework for saddle point problems

Primal-Dual Sequential Subspace Optimization for Saddle-point Problems

A Decentralized Proximal Point-type Method for Saddle Point Problems

A Second Order Primal–Dual Dynamical System for a Convex–Concave Bilinear Saddle Point Problem

A Unified Distributed Method for Constrained Networked Optimization via Saddle-Point Dynamics

Primal-dual Accelerated Mirror-Descent Method for Constrained Bilinear Saddle-Point Problems

A stochastic preconditioned Douglas-Rachford splitting method for saddle-point problems