Abstract:Nonconvex-nonconcave minimax optimization has received intense attention over the last decade due to its broad applications in machine learning. Most existing algorithms rely on one-sided information, such as the convexity (resp. concavity) of the primal (resp. dual) functions, or other specific structures, such as the Polyak-Łojasiewicz (PŁ) and Kurdyka-Łojasiewicz (KŁ) conditions. However, verifying these regularity conditions is challenging in practice. To meet this challenge, we propose a novel universally applicable single-loop algorithm, the doubly smoothed gradient descent ascent method (DS-GDA), which naturally balances the primal and dual updates. That is, DS-GDA with the same hyperparameters is able to uniformly solve nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems with one-sided KŁ properties, achieving convergence with $\mathcal{O}(\epsilon^{-4})$ complexity. Sharper (even optimal) iteration complexity can be obtained when the KŁ exponent is known. Specifically, under the one-sided KŁ condition with exponent $\theta\in(0,1)$, DS-GDA converges with an iteration complexity of $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$. They all match the corresponding best results in the literature. Moreover, we show that DS-GDA is practically applicable to general nonconvex-nonconcave problems even without any regularity conditions, such as the PŁ condition, KŁ condition, or weak Minty variational inequalities condition. For various challenging nonconvex-nonconcave examples in the literature, including ``Forsaken'', ``Bilinearly-coupled minimax'', ``Sixth-order polynomial'', and ``PolarGame'', the proposed DS-GDA can all get rid of limit cycles. To the best of our knowledge, this is the first first-order algorithm to achieve convergence on all of these formidable problems.

Dual Space Preconditioning for Gradient Descent

Some Primal-Dual Theory for Subgradient Methods for Strongly Convex Optimization

Anisotropic Proximal Gradient

Primal-dual proximal bundle and conditional gradient methods for convex problems

Mirror and Preconditioned Gradient Descent in Wasserstein Space

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions

A Preconditioned Version of a Nested Primal-Dual Algorithm for Image Deblurring

Asymptotic Analysis of Conditioned Stochastic Gradient Descent

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Toward a Unified Theory of Gradient Descent under Generalized Smoothness

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Stochastic Gradient Methods with Preconditioned Updates

Intermediate Gradient Methods with Relative Inexactness

Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functions

Anisotropic Proximal Point Algorithm

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

A Nonlinear Bregman Primal-Dual Framework for Optimizing Nonconvex Infimal Convolutions

Exponential Convergence of Augmented Primal-dual Gradient Algorithms for Partially Strongly Convex Functions

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces