Abstract:In this paper, we establish the first explicit and non-asymptotic global convergence analysis of the BFGS method when deployed with an inexact line search scheme that satisfies the Armijo-Wolfe conditions. We show that BFGS achieves a global convergence rate of $(1-\frac{1}{\kappa})^k$ for $\mu$-strongly convex functions with $L$-Lipschitz gradients, where $\kappa=\frac{L}{\mu}$ denotes the condition number. Furthermore, if the objective function's Hessian is Lipschitz, BFGS with the Armijo-Wolfe line search achieves a linear convergence rate only determined by the line search parameters and independent of the condition number. These results hold for any initial point $x_0$ and any symmetric positive definite initial Hessian approximation matrix $B_0$, although the choice of $B_0$ affects the iteration count required to attain these rates. Specifically, we show that for $B_0 = LI$, the rate of $O((1-\frac{1}{\kappa})^k)$ appears from the first iteration, while for $B_0 = \mu I$, it takes $d\log \kappa$ iterations. Conversely, the condition number-independent linear convergence rate for $B_0 = LI$ occurs after $O\left(\kappa\left(d +\frac{M \sqrt{f(x_0)-f(x_*)}}{\mu^{3/2}}\right)\right)$ iterations, whereas for $B_0 = \mu I$, it holds after $O\left(\frac{M \sqrt{f(x_0)-f(x_*)}}{\mu^{3/2}}\left(d\log \kappa + \kappa\right)\right)$ iterations. Here, $d$ denotes the dimension of the problem, $M$ is the Lipschitz parameter of the Hessian, and $x_*$ denotes the optimal solution. We further leverage these global linear convergence results to characterize the overall iteration complexity of BFGS when deployed with the Armijo-Wolfe line search.

Solving a Type of Modifed BFGS Algorithm with any Rank Defects and the Local Q-superlinear Convergence Properties

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

Global convergence of a BFGS-type algorithm for nonconvex multiobjective optimization problems

Global Complexity Analysis of BFGS

Global convergence of a modified BFGS-type method based on function information for nonconvex multiobjective optimization problems

A globally convergent SQP-type method with least constraint violation for nonlinear semidefinite programming

Global Convergence Analysis of a New Nonmonotone BFGS Algorithm on Convex Objective Functions

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

A Modified HS Conjugate Gradient Method and Its Global Convergence

A rank-one fitting algorithm for unconstrained optimization problems

Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search

A Modified Gauss-Newton-based BFGS Method for Symmetric Nonlinear Equations

A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

Two Modified Single-Parameter Scaling Broyden–Fletcher–Goldfarb–Shanno Algorithms for Solving Nonlinear System of Symmetric Equations

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

A globalization of L-BFGS and the Barzilai-Borwein method for nonconvex unconstrained optimization