Abstract:The optimistic gradient method has seen increasing popularity for solving convex-concave saddle point problems. To analyze its iteration complexity, a recent work [<a class="link-https" data-arxiv-id="1906.01115" href="https://arxiv.org/abs/1906.01115">arXiv:1906.01115</a>] proposed an interesting perspective that interprets this method as an approximation to the proximal point method. In this paper, we follow this approach and distill the underlying idea of optimism to propose a generalized optimistic method, which includes the optimistic gradient method as a special case. Our general framework can handle constrained saddle point problems with composite objective functions and can work with arbitrary norms using Bregman distances. Moreover, we develop a backtracking line search scheme to select the step sizes without knowledge of the smoothness coefficients. We instantiate our method with first-, second- and higher-order oracles and give best-known global iteration complexity bounds. For our first-order method, we show that the averaged iterates converge at a rate of $O(1/N)$ when the objective function is convex-concave, and it achieves linear convergence when the objective is strongly-convex-strongly-concave. For our second- and higher-order methods, under the additional assumption that the distance-generating function has Lipschitz gradient, we prove a complexity bound of $O(1/\epsilon^\frac{2}{p+1})$ in the convex-concave setting and a complexity bound of $O((L_pD^\frac{p-1}{2}/\mu)^\frac{2}{p+1}+\log\log\frac{1}{\epsilon})$ in the strongly-convex-strongly-concave setting, where $L_p$ ($p\geq 2$) is the Lipschitz constant of the $p$-th-order derivative, $\mu$ is the strong convexity parameter, and $D$ is the initial Bregman distance to the saddle point. Moreover, our line search scheme provably only requires a constant number of calls to a subproblem solver per iteration on average, making our first- and second-order methods particularly amenable to implementation.

Convexification and existence of a saddle point in a pth-power reformulation for nonconvex constrained optimization

Existence of a Saddle Point in Nonconvex Constrained Optimization

On Saddle Points of Augmented Lagrangians for Constrained Nonconvex Optimization

Nonsmooth convex-concave saddle point problems with cardinality penalties

Constraint Qualifications for Optimality Conditions and Total Lagrange Dualities in Convex Infinite Programming

Stable and Total Fenchel Duality for Convex Optimization Problems in Locally Convex Spaces

A Theoretical Revisit to Linear Convergence for Saddle Point Problems

Convexification, Concavification and Monotonization in Global Optimization

Second Order Optimality Conditions and Reformulations for Nonconvex Quadratically Constrained Quadratic Programming Problems

Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

Towards Strong Duality in Integer Programming

The Saddle-Gradient Method for Linear Constrained Convex Programming

Constraint Qualifications for Convex Inequality Systems with Applications in Constrained Optimization

Equivalent Sufficient Conditions for Global Optimality of Quadratically Constrained Quadratic Program

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Optimization Condition and Algorithm of Optimization with Convertible Nonconvex Function

Optimality Conditions and Complexity for Non-Lipschitz Constrained Optimization Problems

A Second Order Primal–Dual Dynamical System for a Convex–Concave Bilinear Saddle Point Problem

Contractivity and linear convergence in bilinear saddle-point problems: An operator-theoretic approach

On Linear Convergence in Smooth Convex-Concave Bilinearly-Coupled Saddle-Point Optimization: Lower Bounds and Optimal Algorithms

Optimality and Complexity for Constrained Optimization Problems with Nonconvex Regularization.