Abstract:In this paper, we propose a quasi-Newton method for solving smooth and monotone nonlinear equations, including unconstrained minimization and minimax optimization as special cases. For the strongly monotone setting, we establish two global convergence bounds: (i) a linear convergence rate that matches the rate of the celebrated extragradient method, and (ii) an explicit global superlinear convergence rate that provably surpasses the linear convergence rate after at most ${O}(d)$ iterations, where $d$ is the problem's dimension. In addition, for the case where the operator is only monotone, we prove a global convergence rate of ${O}(\min\{{1}/{k},{\sqrt{d}}/{k^{1.25}}\})$ in terms of the duality gap. This matches the rate of the extragradient method when $k = {O}(d^2)$ and is faster when $k = \Omega(d^2)$. These results are the first global convergence results to demonstrate a provable advantage of a quasi-Newton method over the extragradient method, without querying the Jacobian of the operator. Unlike classical quasi-Newton methods, we achieve this by using the hybrid proximal extragradient framework and a novel online learning approach for updating the Jacobian approximation matrices. Specifically, guided by the convergence analysis, we formulate the Jacobian approximation update as an online convex optimization problem over non-symmetric matrices, relating the regret of the online problem to the convergence rate of our method. To facilitate efficient implementation, we further develop a tailored online learning algorithm based on an approximate separation oracle, which preserves structures such as symmetry and sparsity in the Jacobian matrices.

A New Three-Term Spectral Conjugate Gradient Algorithm with Higher Numerical Performance for Solving Large Scale Optimization Problems Based on Quasi-Newton Equation.

A New Three-Term Conjugate Gradient Algorithm with Modified Gradient-Differences for Solving Unconstrained Optimization Problems

The convergence of conjugate gradient method with nonmonotone line search

A class of new three-term descent conjugate gradient algorithms for large-scale unconstrained optimization and applications to image restoration problems

Two sufficient descent three-term conjugate gradient methods for unconstrained optimization problems with applications in compressive sensing

A spectral three-term Hestenes–Stiefel conjugate gradient method

Spectral conjugate gradient methods for vector optimization problems

A hybrid three-term conjugate gradient projection method for constrained nonlinear monotone equations with applications

A New Version Coefficient Of Three-Term Conjugate Gradient Method To Solve Unconstrained Optimization

Optimization of unconstrained problems using a developed algorithm of spectral conjugate gradient method calculation

Some three-term conjugate gradient methods with the new direction structure

A three-term conjugate gradient descent method with some applications

A convergent hybrid three-term conjugate gradient method with sufficient descent property for unconstrained optimization

Solving Large-Scale Unconstrained Optimization Problems with an Efficient Conjugate Gradient Class

A family of spectral conjugate gradient method with strong convergence and its applications in image restoration and machine learning

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

A New Hybrid Three-Term LS-CD Conjugate Gradient In Solving Unconstrained Optimization Problems

A Modified Nonlinear Conjugate Gradient Algorithm for Large-Scale Nonsmooth Convex Optimization

A New Nonmonotone Spectral Projected Gradient Algorithm for Box-Constrained Optimization Problems in M×n Real Matrix Space with Application in Image Clustering.