Abstract:Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O$((1/\sqrt{t})^t)$. The methods that obtain this rate, however, exhibit a well-known drawback: they require the storage of the previous Hessian approximation matrix or all past curvature information to form the current Hessian inverse approximation. Limited-memory variants of quasi-Newton methods such as the celebrated L-BFGS alleviate this issue by leveraging a limited window of past curvature information to construct the Hessian inverse approximation. As a result, their per iteration complexity and storage requirement is O$(\tau d)$ where $\tau\le d$ is the size of the window and $d$ is the problem dimension reducing the O$(d^2)$ computational cost and memory requirement of standard quasi-Newton methods. However, to the best of our knowledge, there is no result showing a non-asymptotic superlinear convergence rate for any limited-memory quasi-Newton method. In this work, we close this gap by presenting a Limited-memory Greedy BFGS (LG-BFGS) method that can achieve an explicit non-asymptotic superlinear rate. We incorporate displacement aggregation, i.e., decorrelating projection, in post-processing gradient variations, together with a basis vector selection scheme on variable variations, which greedily maximizes a progress measure of the Hessian estimate to the true Hessian. Their combination allows past curvature information to remain in a sparse subspace while yielding a valid representation of the full history. Interestingly, our established non-asymptotic superlinear convergence rate demonstrates an explicit trade-off between the convergence speed and memory requirement, which to our knowledge, is the first of its kind. Numerical results corroborate our theoretical findings and demonstrate the effectiveness of our method.

On finite termination of quasi-Newton methods on quadratic problems

On finite termination of quasi-Newton methods on quadratic problems

Quasi-Newton methods for minimizing a quadratic function subject to uncertainty

Online Learning Guided Quasi-Newton Methods with Global Non-Asymptotic Convergence

A mechanism of three-dimensional quadratic termination for the gradient method with applications

Finite Termination of Inexact Proximal Point Algorithms in Hilbert Spaces

Online Learning Guided Curvature Approximation: A Quasi-Newton Method with Global Non-Asymptotic Superlinear Convergence

Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate

Approximating Hessian matrices using Bayesian inference: a new approach for quasi-Newton methods in stochastic optimization

Soft quasi-Newton: Guaranteed positive definiteness by relaxing the secant constraint

A New Diagonal Quasi-Newton Updating Method With Scaled Forward Finite Differences Directional Derivative for Unconstrained Optimization

A method to guarantee local convergence for sequential quadratic programming with poor Hessian approximation

Adaptive Greedy Quasi-Newton with Superlinear Rate and Global Convergence Guarantee

Novel Limited Memory Quasi-Newton Methods Based On Optimal Matrix Approximation

Adaptive regularized quasi-newton method using inexact first-order information

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

On the Performance of SQP Methods for Nonlinear Optimization

Quasi-Newton approaches to Interior Point Methods for quadratic problems

Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

A nonmonotone proximal quasi-Newton method for multiobjective optimization