Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

Newton Method Revisited: Global Convergence Rates up to $\mathcal {O}\left(k^{-3} \right)$ for Stepsize Schedules and Linesearch Procedures

The convergence of conjugate gradient method with nonmonotone line search

Multilevel Regularized Newton Methods with Fast Convergence Rates

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Adaptive Greedy Quasi-Newton with Superlinear Rate and Global Convergence Guarantee

Greedy Newton: Newton's Method with Exact Line Search

Super-Universal Regularized Newton Method

Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

Sketch-and-Project Meets Newton Method: Global $\mathcal O(k^{-2})$ Convergence with Low-Rank Updates

Stochastic Newton Proximal Extragradient Method

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Yet another fast variant of Newton’s method for nonconvex optimization

Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedule

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

A Communication-Efficient Decentralized Newton's Method with Provably Faster Convergence

On a globally convergent semismooth* Newton method in nonsmooth nonconvex optimzation