Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

An accelerated adaptive two-step Levenberg–Marquardt method with the modified Metropolis criterion

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

An Enhanced Levenberg--Marquardt Method via Gram Reduction

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A two-step Broyden-like method for nonlinear equations

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

A New Fast Adaptive Linearized Alternating Direction Multiplier Method for Convex Optimization

A Modified Dai–Liao Conjugate Gradient Method Based on a Scalar Matrix Approximation of Hessian and Its Application

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

Higher-Order Corrections to Optimisers based on Newton's Method

Gradient Methods with Adaptive Step-Sizes

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Adaptive Greedy Quasi-Newton with Superlinear Rate and Global Convergence Guarantee

An adaptive linearized alternating direction multiplier method with a relaxation step for convex programming

The Levenberg–Marquardt method: an overview of modern convergence theories and more

On the acceleration of the Barzilai-Borwein method

Quadratic Convergence of Levenberg-Marquardt Method for Elliptic and Parabolic Inverse Robin Problems

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Two-Step Fifth-Order Efficient Jacobian-Free Iterative Method for Solving Nonlinear Systems