Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations

Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods

Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence.

On the Convergence of the Accelerated Riccati Iteration Method.

Explicit Superlinear Convergence Rates of The SR1 Algorithm

A two-step Broyden-like method for nonlinear equations

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

On a non-archimedean broyden method

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

Global non-asymptotic super-linear convergence rates of regularized proximal quasi-Newton methods on non-smooth composite problems

Towards Explicit Superlinear Convergence Rate for SR1

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

On the local convergence of the semismooth Newton method for composite optimization

Incremental Gauss--Newton Methods with Superlinear Convergence Rates

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.