Abstract:Nonconvex-nonconcave minimax optimization has received intense attention over the last decade due to its broad applications in machine learning. Most existing algorithms rely on one-sided information, such as the convexity (resp. concavity) of the primal (resp. dual) functions, or other specific structures, such as the Polyak-Łojasiewicz (PŁ) and Kurdyka-Łojasiewicz (KŁ) conditions. However, verifying these regularity conditions is challenging in practice. To meet this challenge, we propose a novel universally applicable single-loop algorithm, the doubly smoothed gradient descent ascent method (DS-GDA), which naturally balances the primal and dual updates. That is, DS-GDA with the same hyperparameters is able to uniformly solve nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems with one-sided KŁ properties, achieving convergence with $\mathcal{O}(\epsilon^{-4})$ complexity. Sharper (even optimal) iteration complexity can be obtained when the KŁ exponent is known. Specifically, under the one-sided KŁ condition with exponent $\theta\in(0,1)$, DS-GDA converges with an iteration complexity of $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$. They all match the corresponding best results in the literature. Moreover, we show that DS-GDA is practically applicable to general nonconvex-nonconcave problems even without any regularity conditions, such as the PŁ condition, KŁ condition, or weak Minty variational inequalities condition. For various challenging nonconvex-nonconcave examples in the literature, including ``Forsaken'', ``Bilinearly-coupled minimax'', ``Sixth-order polynomial'', and ``PolarGame'', the proposed DS-GDA can all get rid of limit cycles. To the best of our knowledge, this is the first first-order algorithm to achieve convergence on all of these formidable problems.

Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization

Parameter-free accelerated gradient descent for nonconvex minimization

Optimal and parameter-free gradient minimization methods for convex and nonconvex optimization

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O(<SUP>-7/4</SUP>) Complexity

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the $O(\epsilon^{-7/4})$ Complexity

Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O(ε-7/4) Complexity.

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems

Non-Uniform Smoothness for Gradient Descent

Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization

Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problems

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

A Proximal-Gradient Method for Constrained Optimization

On a Family of Relaxed Gradient Descent Methods for Quadratic Minimization

AGDA+: Proximal Alternating Gradient Descent Ascent Method With a Nonmonotone Adaptive Step-Size Search For Nonconvex Minimax Problems

Fast Gradient Algorithm with Dry-like Friction and Nonmonotone Line Search for Nonconvex Optimization Problems

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

A Local Nearly Linearly Convergent First-Order Method for Nonsmooth Functions with Quadratic Growth