Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

On a New Updating Rule of the Levenberg–Marquardt Parameter

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

The convergence of conjugate gradient method with nonmonotone line search

An accelerated adaptive two-step Levenberg–Marquardt method with the modified Metropolis criterion

A Trust Region-Type Method for Solving Monotone Variational Inequality

An Enhanced Levenberg--Marquardt Method via Gram Reduction

Adaptive Levenberg–Marquardt Algorithm: A New Optimization Strategy for Levenberg–Marquardt Neural Networks

Riemannian Levenberg-Marquardt Method with Global and Local Convergence Properties

The Levenberg–Marquardt method: an overview of modern convergence theories and more

Improving Levenberg-Marquardt Algorithm for Neural Networks

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Convergence analysis of Levenberg-Marquardt method with Singular Scaling for nonzero residue nonlinear least-squares problems

Global Convergence of a Stochastic Levenberg–Marquardt Algorithm Based on Trust Region

A two-step Broyden-like method for nonlinear equations

On inertial Levenberg-Marquardt type methods for solving nonlinear ill-posed operator equations

Complexity bound of a Levenberg–Marquardt algorithm based on probabilistic Jacobian models

Higher-Order Corrections to Optimisers based on Newton's Method

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

An Efficient Modified Residual-Based Algorithm for Large Scale Symmetric Nonlinear Equations by Approximating Successive Iterated Gradients

A new algorithm for solving nonlinear parabolic equations using extreme learning machine method with parameter retention