Abstract:Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O$((1/\sqrt{t})^t)$. The methods that obtain this rate, however, exhibit a well-known drawback: they require the storage of the previous Hessian approximation matrix or all past curvature information to form the current Hessian inverse approximation. Limited-memory variants of quasi-Newton methods such as the celebrated L-BFGS alleviate this issue by leveraging a limited window of past curvature information to construct the Hessian inverse approximation. As a result, their per iteration complexity and storage requirement is O$(\tau d)$ where $\tau\le d$ is the size of the window and $d$ is the problem dimension reducing the O$(d^2)$ computational cost and memory requirement of standard quasi-Newton methods. However, to the best of our knowledge, there is no result showing a non-asymptotic superlinear convergence rate for any limited-memory quasi-Newton method. In this work, we close this gap by presenting a Limited-memory Greedy BFGS (LG-BFGS) method that can achieve an explicit non-asymptotic superlinear rate. We incorporate displacement aggregation, i.e., decorrelating projection, in post-processing gradient variations, together with a basis vector selection scheme on variable variations, which greedily maximizes a progress measure of the Hessian estimate to the true Hessian. Their combination allows past curvature information to remain in a sparse subspace while yielding a valid representation of the full history. Interestingly, our established non-asymptotic superlinear convergence rate demonstrates an explicit trade-off between the convergence speed and memory requirement, which to our knowledge, is the first of its kind. Numerical results corroborate our theoretical findings and demonstrate the effectiveness of our method.

Explicit Superlinear Convergence Rates of The SR1 Algorithm

Towards Explicit Superlinear Convergence Rate for SR1

Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods

Global non-asymptotic super-linear convergence rates of regularized proximal quasi-Newton methods on non-smooth composite problems

Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence.

Explicit Superlinear Convergence Rates of Broyden's Methods in Nonlinear Equations

Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

Explicit Superlinear Convergence of Broyden's Method in Nonlinear Equations

Convergence Criterion of Newton's Method for Singular Systems with Constant Rank Derivatives

On a Faster $R$-Linear Convergence Rate of the Barzilai-Borwein Method

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate

Strong convergence of an explicit numerical approximation for n -dimensional superlinear SDEs with positive solutions

Greedy and Random Broyden's Methods with Explicit Superlinear Convergence Rates in Nonlinear Equations

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Linear Convergence of Adaptive Stochastic Gradient Descent

Linear convergence of first order methods for non-strongly convex optimization

A stabilized SQP method: superlinear convergence

On the Complexity Analysis of the Primal Solutions for the Accelerated Randomized Dual Coordinate Ascent

Linear Convergence of Variance-Reduced Stochastic Gradient without Strong Convexity