Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

A two-step Broyden-like method for nonlinear equations

The convergence of conjugate gradient method with nonmonotone line search

Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations

Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

Two Modified Single-Parameter Scaling Broyden–Fletcher–Goldfarb–Shanno Algorithms for Solving Nonlinear System of Symmetric Equations

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

On a non-archimedean broyden method

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

A derivative-free memoryless BFGS hyperplane projection method for solving large-scale nonlinear monotone equations

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Two-Step Fifth-Order Efficient Jacobian-Free Iterative Method for Solving Nonlinear Systems

A new approximate descent derivative-free algorithm for large-scale nonlinear symmetric equations

A Bregman-Kaczmarz method for nonlinear systems of equations

Nonlinearly preconditioned semismooth Newton algorithms for nonlinear nonsmooth systems

The complex-step Newton method and its convergence

An accelerated adaptive two-step Levenberg–Marquardt method with the modified Metropolis criterion

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Barzilai and Borwein conjugate gradient method equipped with a non-monotone line search technique and its application on non-negative matrix factorization

New high-order convergence iteration methods without employing derivatives for solving nonlinear equations