Abstract:We present a modified limited memory BFGS (L-BFGS) method that converges globally and linearly for nonconvex objective functions. Its distinguishing feature is that it turns into L-BFGS if the iterates cluster at a point near which the objective is strongly convex with Lipschitz gradients, thereby inheriting the outstanding effectiveness of the classical method. These strong convergence guarantees are enabled by a novel form of cautious updating, where, among others, it is decided anew in each iteration which of the stored pairs are used for updating and which ones are skipped. In particular, this yields the first modification of cautious updating for which all cluster points are stationary while the spectrum of the L-BFGS operator is not permanently restricted, and this holds without Lipschitz continuity of the gradient. In fact, for Wolfe-Powell line searches we show that continuity of the gradient is sufficient for global convergence, which extends to other descent methods. Since we allow the memory size to be zero in the globalized L-BFGS method, we also obtain a new globalization of the Barzilai-Borwein spectral gradient (BB) method. The convergence analysis is developed in Hilbert space under comparably weak assumptions and covers Armijo and Wolfe-Powell line searches. We illustrate the theoretical findings with numerical experiments. The experiments indicate that if one of the parameters of the cautious updating is chosen sufficiently small, then the modified method agrees entirely with L-BFGS/BB. We also discuss this in the theoretical part. An implementation of the new method is available on arXiv.

Globalization of convergence of the constrained piecewise Levenberg–Marquardt method

The convergence of conjugate gradient method with nonmonotone line search

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

The Levenberg–Marquardt method: an overview of modern convergence theories and more

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

A globalization of L-BFGS and the Barzilai-Borwein method for nonconvex unconstrained optimization

Linearly Constrained Global Optimization Via Piecewise-Linear Approximation

Global convergence of Riemannian line search methods with a Zhang-Hager-type condition

A Smoothing Levenberg-Marquardt Type Method for Lcp

Local and Global Behavior for Algorithms of Solving Equations

Global Convergence of the Polak-Ribière and Hestenes-Stiefel Conjugate Gradient Methods for the Unconstrained Nonlinear Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

A fresh look at nonsmooth Levenberg–Marquardt methods with applications to bilevel optimization

A global and superlinearly convergent algorithm for nonlinear nondifferential convex programming problems with a generalized Armijo line-search

A Nonmonotone Filter Barzilai-Borwein Method for Optimization

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

A New Adaptive Levenberg–Marquardt Method for Nonlinear Equations and Its Convergence Rate under the Hölderian Local Error Bound Condition

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization