Abstract:In this paper, we establish the first explicit and non-asymptotic global convergence analysis of the BFGS method when deployed with an inexact line search scheme that satisfies the Armijo-Wolfe conditions. We show that BFGS achieves a global convergence rate of $(1-\frac{1}{\kappa})^k$ for $\mu$-strongly convex functions with $L$-Lipschitz gradients, where $\kappa=\frac{L}{\mu}$ denotes the condition number. Furthermore, if the objective function's Hessian is Lipschitz, BFGS with the Armijo-Wolfe line search achieves a linear convergence rate only determined by the line search parameters and independent of the condition number. These results hold for any initial point $x_0$ and any symmetric positive definite initial Hessian approximation matrix $B_0$, although the choice of $B_0$ affects the iteration count required to attain these rates. Specifically, we show that for $B_0 = LI$, the rate of $O((1-\frac{1}{\kappa})^k)$ appears from the first iteration, while for $B_0 = \mu I$, it takes $d\log \kappa$ iterations. Conversely, the condition number-independent linear convergence rate for $B_0 = LI$ occurs after $O\left(\kappa\left(d +\frac{M \sqrt{f(x_0)-f(x_*)}}{\mu^{3/2}}\right)\right)$ iterations, whereas for $B_0 = \mu I$, it holds after $O\left(\frac{M \sqrt{f(x_0)-f(x_*)}}{\mu^{3/2}}\left(d\log \kappa + \kappa\right)\right)$ iterations. Here, $d$ denotes the dimension of the problem, $M$ is the Lipschitz parameter of the Hessian, and $x_*$ denotes the optimal solution. We further leverage these global linear convergence results to characterize the overall iteration complexity of BFGS when deployed with the Armijo-Wolfe line search.

New Cautious BFGS Algorithm Based on Modified Armijo-type Line Search

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

Non-asymptotic Global Convergence Analysis of BFGS with the Armijo-Wolfe Line Search

Global Complexity Analysis of BFGS

The convergence of conjugate gradient method with nonmonotone line search

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

A class of nonmonotone Armijo-type line search method for unconstrained optimization

Combined BFGS-chaos method for solving geometric constraint

Global Convergence of BFGS and PRP Methods under a Modified Weak Wolfe–Powell Line Search

A globalization of L-BFGS and the Barzilai-Borwein method for nonconvex unconstrained optimization

A New Backtracking Inexact Bfgs Method for Symmetric Nonlinear Equations

A Line Search Multigrid Method for Large-Scale Nonlinear Optimization

A Barzilai-Borwein Descent Method for Multiobjective Optimization Problems

An improvement of the Goldstein line search

New Trust Region Method for Line Search

Global convergence of a BFGS-type algorithm for nonconvex multiobjective optimization problems

Analysis of the gradient method with an Armijo–Wolfe line search on a class of non-smooth convex functions

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

Two Modified Single-Parameter Scaling Broyden–Fletcher–Goldfarb–Shanno Algorithms for Solving Nonlinear System of Symmetric Equations