Abstract:The optimistic gradient method has seen increasing popularity for solving convex-concave saddle point problems. To analyze its iteration complexity, a recent work [<a class="link-https" data-arxiv-id="1906.01115" href="https://arxiv.org/abs/1906.01115">arXiv:1906.01115</a>] proposed an interesting perspective that interprets this method as an approximation to the proximal point method. In this paper, we follow this approach and distill the underlying idea of optimism to propose a generalized optimistic method, which includes the optimistic gradient method as a special case. Our general framework can handle constrained saddle point problems with composite objective functions and can work with arbitrary norms using Bregman distances. Moreover, we develop a backtracking line search scheme to select the step sizes without knowledge of the smoothness coefficients. We instantiate our method with first-, second- and higher-order oracles and give best-known global iteration complexity bounds. For our first-order method, we show that the averaged iterates converge at a rate of $O(1/N)$ when the objective function is convex-concave, and it achieves linear convergence when the objective is strongly-convex-strongly-concave. For our second- and higher-order methods, under the additional assumption that the distance-generating function has Lipschitz gradient, we prove a complexity bound of $O(1/\epsilon^\frac{2}{p+1})$ in the convex-concave setting and a complexity bound of $O((L_pD^\frac{p-1}{2}/\mu)^\frac{2}{p+1}+\log\log\frac{1}{\epsilon})$ in the strongly-convex-strongly-concave setting, where $L_p$ ($p\geq 2$) is the Lipschitz constant of the $p$-th-order derivative, $\mu$ is the strong convexity parameter, and $D$ is the initial Bregman distance to the saddle point. Moreover, our line search scheme provably only requires a constant number of calls to a subproblem solver per iteration on average, making our first- and second-order methods particularly amenable to implementation.

Saddle Point Optimization with Approximate Minimization Oracle and Its Application to Robust Berthing Control

Saddle point optimization with approximate minimization oracle

Oracle-Based Robust Optimization via Online Learning

Application of optimal control theory based on the evolution strategy (CMA-ES) to automatic berthing (part: 2)

Actively Learning Combinatorial Optimization Using a Membership Oracle

New Multi-Objective Optimization Control and Its Application to Nonlinear Evaporation Process

Optimization on planning of trajectory and control of autonomous berthing and unberthing for the realistic port geometry

Controllable robust optimization of ship concept design for quantifying uncertainty of marine water quality forecasts

Generalized Optimistic Methods for Convex-Concave Saddle Point Problems

A Max-Min-Max Algorithm for Large-Scale Robust Optimization

Proximal Oracles for Optimization and Sampling

Gradient Methods with Dynamic Inexact Oracles

A Frank-Wolfe Algorithm for Oracle-based Robust Optimization

Acceleration with a Ball Optimization Oracle

Stochastic Optimization Algorithms for Problems with Controllable Biased Oracles

Consensus-Based Optimization for Saddle Point Problems

A Geometric Branch and Bound Method for a Class of Robust Maximization Problems of Convex Functions

High Probability Complexity Bounds for Adaptive Step Search Based on Stochastic Oracles

Approximation of the objective insensitivity regions using Hierarchic Memetic Strategy coupled with Covariance Matrix Adaptation Evolutionary Strategy

Multi-Objective Optimization Method for the Transverse Section of A Midship Based on Robustness

Switch and Conquer: Efficient Algorithms By Switching Stochastic Gradient Oracles For Decentralized Saddle Point Problems