Abstract:Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O$((1/\sqrt{t})^t)$. The methods that obtain this rate, however, exhibit a well-known drawback: they require the storage of the previous Hessian approximation matrix or all past curvature information to form the current Hessian inverse approximation. Limited-memory variants of quasi-Newton methods such as the celebrated L-BFGS alleviate this issue by leveraging a limited window of past curvature information to construct the Hessian inverse approximation. As a result, their per iteration complexity and storage requirement is O$(\tau d)$ where $\tau\le d$ is the size of the window and $d$ is the problem dimension reducing the O$(d^2)$ computational cost and memory requirement of standard quasi-Newton methods. However, to the best of our knowledge, there is no result showing a non-asymptotic superlinear convergence rate for any limited-memory quasi-Newton method. In this work, we close this gap by presenting a Limited-memory Greedy BFGS (LG-BFGS) method that can achieve an explicit non-asymptotic superlinear rate. We incorporate displacement aggregation, i.e., decorrelating projection, in post-processing gradient variations, together with a basis vector selection scheme on variable variations, which greedily maximizes a progress measure of the Hessian estimate to the true Hessian. Their combination allows past curvature information to remain in a sparse subspace while yielding a valid representation of the full history. Interestingly, our established non-asymptotic superlinear convergence rate demonstrates an explicit trade-off between the convergence speed and memory requirement, which to our knowledge, is the first of its kind. Numerical results corroborate our theoretical findings and demonstrate the effectiveness of our method.

An Accelerated Linearly Convergent Stochastic L-BFGS Algorithm

Accelerated Stochastic ADMM with Variance Reduction

The convergence of conjugate gradient method with nonmonotone line search

A globalization of L-BFGS and the Barzilai-Borwein method for nonconvex unconstrained optimization

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

Accelerated Variance Reduction Stochastic ADMM for Large-Scale Machine Learning

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning

R-Linear Convergence of Limited Memory Steepest Descent

Fast Asynchronous Parallel Stochastic Gradient Descent: A Lock-Free Approach With Convergence Guarantee

Asynchronous Parallel Stochastic Quasi-Newton Methods

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

A New Fast Adaptive Linearized Alternating Direction Multiplier Method for Convex Optimization

Parallel L-BFGS-B Algorithm on GPU.

Asynchronous Accelerated Stochastic Gradient Descent.

Accelerated stochastic admm for empirical risk minimization

Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate

Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence

Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning