Abstract:In this paper, we establish the first explicit and non-asymptotic global convergence analysis of the BFGS method when deployed with an inexact line search scheme that satisfies the Armijo-Wolfe conditions. We show that BFGS achieves a global convergence rate of $(1-\frac{1}{\kappa})^k$ for $\mu$-strongly convex functions with $L$-Lipschitz gradients, where $\kappa=\frac{L}{\mu}$ denotes the condition number. Furthermore, if the objective function's Hessian is Lipschitz, BFGS with the Armijo-Wolfe line search achieves a linear convergence rate only determined by the line search parameters and independent of the condition number. These results hold for any initial point $x_0$ and any symmetric positive definite initial Hessian approximation matrix $B_0$, although the choice of $B_0$ affects the iteration count required to attain these rates. Specifically, we show that for $B_0 = LI$, the rate of $O((1-\frac{1}{\kappa})^k)$ appears from the first iteration, while for $B_0 = \mu I$, it takes $d\log \kappa$ iterations. Conversely, the condition number-independent linear convergence rate for $B_0 = LI$ occurs after $O\left(\kappa\left(d +\frac{M \sqrt{f(x_0)-f(x_*)}}{\mu^{3/2}}\right)\right)$ iterations, whereas for $B_0 = \mu I$, it holds after $O\left(\frac{M \sqrt{f(x_0)-f(x_*)}}{\mu^{3/2}}\left(d\log \kappa + \kappa\right)\right)$ iterations. Here, $d$ denotes the dimension of the problem, $M$ is the Lipschitz parameter of the Hessian, and $x_*$ denotes the optimal solution. We further leverage these global linear convergence results to characterize the overall iteration complexity of BFGS when deployed with the Armijo-Wolfe line search.

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

The convergence of conjugate gradient method with nonmonotone line search

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

Global Complexity Analysis of BFGS

Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search

Global convergence of a BFGS-type algorithm for nonconvex multiobjective optimization problems

A globalization of L-BFGS and the Barzilai-Borwein method for nonconvex unconstrained optimization

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Global convergence of a modified BFGS-type method based on function information for nonconvex multiobjective optimization problems

A Line Search Multigrid Method for Large-Scale Nonlinear Optimization

BFGS convergence to nonsmooth minimizers of convex functions

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

Global Convergence of the Polak-Ribière and Hestenes-Stiefel Conjugate Gradient Methods for the Unconstrained Nonlinear Optimization

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

A globally convergent SQP-type method with least constraint violation for nonlinear semidefinite programming

A global and superlinearly convergent algorithm for nonlinear nondifferential convex programming problems with a generalized Armijo line-search

Coderivative-Based Newton Methods with Wolfe Linesearch for Nonsmooth Optimization

Convergence and optimality of an adaptive modified weak Galerkin finite element method