Abstract:Nonconvex-nonconcave minimax optimization has received intense attention over the last decade due to its broad applications in machine learning. Most existing algorithms rely on one-sided information, such as the convexity (resp. concavity) of the primal (resp. dual) functions, or other specific structures, such as the Polyak-Łojasiewicz (PŁ) and Kurdyka-Łojasiewicz (KŁ) conditions. However, verifying these regularity conditions is challenging in practice. To meet this challenge, we propose a novel universally applicable single-loop algorithm, the doubly smoothed gradient descent ascent method (DS-GDA), which naturally balances the primal and dual updates. That is, DS-GDA with the same hyperparameters is able to uniformly solve nonconvex-concave, convex-nonconcave, and nonconvex-nonconcave problems with one-sided KŁ properties, achieving convergence with $\mathcal{O}(\epsilon^{-4})$ complexity. Sharper (even optimal) iteration complexity can be obtained when the KŁ exponent is known. Specifically, under the one-sided KŁ condition with exponent $\theta\in(0,1)$, DS-GDA converges with an iteration complexity of $\mathcal{O}(\epsilon^{-2\max\{2\theta,1\}})$. They all match the corresponding best results in the literature. Moreover, we show that DS-GDA is practically applicable to general nonconvex-nonconcave problems even without any regularity conditions, such as the PŁ condition, KŁ condition, or weak Minty variational inequalities condition. For various challenging nonconvex-nonconcave examples in the literature, including ``Forsaken'', ``Bilinearly-coupled minimax'', ``Sixth-order polynomial'', and ``PolarGame'', the proposed DS-GDA can all get rid of limit cycles. To the best of our knowledge, this is the first first-order algorithm to achieve convergence on all of these formidable problems.

Convergence towards a local minimum by direct search methods with a covering step

The convergence of conjugate gradient method with nonmonotone line search

Convergence depth control for interior point methods

Revisiting Theoretical Guarantees of Direct-Search Methods

A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

Aggressive Local Search for Constrained Optimal Control Problems with Many Local Minima

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

A Combined Global And Local Search Method To Deal With Constrained Optimization For Continuous Tabu Search

Convergence Depth Control for Process System Optimization

A Direct Searching Method for Nonlinear Optimizing Problems and Its Convergence Proof

A New Direct Optimization Method

Effective Front-Descent Algorithms with Convergence Guarantees

A Locally-Biased form of the DIRECT Algorithm

A Smoothing Direct Search Method for Monte Carlo-Based Bound Constrained Composite Nonsmooth Optimization.

Full-Low Evaluation Methods For Bound and Linearly Constrained Derivative-Free Optimization

Local convergence of primal-dual interior point methods for nonlinear semidefinite optimization using the Monteiro-Tsuchiya family of search directions

Beyond Discretization: Learning the Optimal Solution Path

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

Convergence and concentration properties of constant step-size SGD through Markov chains

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Direct search for stochastic optimization in random subspaces with zeroth-, first-, and second-order convergence and expected complexity