Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations

Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods

On the Convergence of the Accelerated Riccati Iteration Method.

Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence.

Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

A two-step Broyden-like method for nonlinear equations

The convergence of conjugate gradient method with nonmonotone line search

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

On a non-archimedean broyden method

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

On the local convergence of the semismooth Newton method for composite optimization

Explicit Superlinear Convergence Rates of The SR1 Algorithm

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Incremental Gauss--Newton Methods with Superlinear Convergence Rates

Global non-asymptotic super-linear convergence rates of regularized proximal quasi-Newton methods on non-smooth composite problems

Local Convergence of Inexact Methods under the Hölder Condition

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

Towards Explicit Superlinear Convergence Rate for SR1