Abstract:The optimistic gradient method has seen increasing popularity for solving convex-concave saddle point problems. To analyze its iteration complexity, a recent work [<a class="link-https" data-arxiv-id="1906.01115" href="https://arxiv.org/abs/1906.01115">arXiv:1906.01115</a>] proposed an interesting perspective that interprets this method as an approximation to the proximal point method. In this paper, we follow this approach and distill the underlying idea of optimism to propose a generalized optimistic method, which includes the optimistic gradient method as a special case. Our general framework can handle constrained saddle point problems with composite objective functions and can work with arbitrary norms using Bregman distances. Moreover, we develop a backtracking line search scheme to select the step sizes without knowledge of the smoothness coefficients. We instantiate our method with first-, second- and higher-order oracles and give best-known global iteration complexity bounds. For our first-order method, we show that the averaged iterates converge at a rate of $O(1/N)$ when the objective function is convex-concave, and it achieves linear convergence when the objective is strongly-convex-strongly-concave. For our second- and higher-order methods, under the additional assumption that the distance-generating function has Lipschitz gradient, we prove a complexity bound of $O(1/\epsilon^\frac{2}{p+1})$ in the convex-concave setting and a complexity bound of $O((L_pD^\frac{p-1}{2}/\mu)^\frac{2}{p+1}+\log\log\frac{1}{\epsilon})$ in the strongly-convex-strongly-concave setting, where $L_p$ ($p\geq 2$) is the Lipschitz constant of the $p$-th-order derivative, $\mu$ is the strong convexity parameter, and $D$ is the initial Bregman distance to the saddle point. Moreover, our line search scheme provably only requires a constant number of calls to a subproblem solver per iteration on average, making our first- and second-order methods particularly amenable to implementation.

Zeroth-Order Negative Curvature Finding: Escaping Saddle Points Without Gradients.

Escape Saddle Points by a Simple Gradient-Descent Based Algorithm

Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently

Escaping saddle points in zeroth-order optimization: the power of two-point estimators

Gradient-Free Method for Heavily Constrained Nonconvex Optimization

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization

First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians

Quantum algorithm for finding the negative curvature direction in non-convex optimization

Efficient First Order Method for Saddle Point Problems with Higher Order Smoothness

Yet another fast variant of Newton’s method for nonconvex optimization

Exploiting Negative Curvature in Conjunction with Adaptive Sampling: Theoretical Results and a Practical Algorithm

Exit Time Analysis for Approximations of Gradient Descent Trajectories Around Saddle Points

Detecting negative eigenvalues of exact and approximate Hessian matrices in optimization

A hybrid inexact regularized Newton and negative curvature method

Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization

Zeroth-Order primal-dual Alternating Projection Gradient Algorithms for Nonconvex Minimax Problems with Coupled linear Constraints

Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

Efficient Saddle Point Evasion and Local Minima Escape in High-Dimensional Non-Convex Optimization

Safe Zeroth-Order Optimization Using Quadratic Local Approximations

Obtaining Lower Query Complexities Through Lightweight Zeroth-Order Proximal Gradient Algorithms