Abstract:Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O$((1/\sqrt{t})^t)$. The methods that obtain this rate, however, exhibit a well-known drawback: they require the storage of the previous Hessian approximation matrix or all past curvature information to form the current Hessian inverse approximation. Limited-memory variants of quasi-Newton methods such as the celebrated L-BFGS alleviate this issue by leveraging a limited window of past curvature information to construct the Hessian inverse approximation. As a result, their per iteration complexity and storage requirement is O$(\tau d)$ where $\tau\le d$ is the size of the window and $d$ is the problem dimension reducing the O$(d^2)$ computational cost and memory requirement of standard quasi-Newton methods. However, to the best of our knowledge, there is no result showing a non-asymptotic superlinear convergence rate for any limited-memory quasi-Newton method. In this work, we close this gap by presenting a Limited-memory Greedy BFGS (LG-BFGS) method that can achieve an explicit non-asymptotic superlinear rate. We incorporate displacement aggregation, i.e., decorrelating projection, in post-processing gradient variations, together with a basis vector selection scheme on variable variations, which greedily maximizes a progress measure of the Hessian estimate to the true Hessian. Their combination allows past curvature information to remain in a sparse subspace while yielding a valid representation of the full history. Interestingly, our established non-asymptotic superlinear convergence rate demonstrates an explicit trade-off between the convergence speed and memory requirement, which to our knowledge, is the first of its kind. Numerical results corroborate our theoretical findings and demonstrate the effectiveness of our method.

A LIMITED-MEMORY PROJECTION METHOD FOR VARIATIONAL INEQUALITY PROBLEMS

A Trust Region-Type Method for Solving Monotone Variational Inequality

Projection and contraction method with double inertial steps for quasi-monotone variational inequalities

Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization

Modified Projection Method For General Variational Inequalities

Projection iterative method for solving general variational inequalities

A limited memory subspace minimization conjugate gradient algorithm for unconstrained optimization

Beyond Monotone Variational Inequalities: Solution Methods and Iteration Complexities

On the Iteration Complexity of Some Projection Methods for Monotone Linear Variational Inequalities

On Approximating Solutions to Non-monotone Variational Inequality Problems: An Approach Through the Modified Projection and Contraction Method

A Modified Form of Inertial Viscosity Projection Methods for Variational Inequality and Fixed Point Problems

A regularized limited memory subspace minimization conjugate gradient method for unconstrained optimization

A family of limited memory three term conjugate gradient methods

An improved relaxed inertial projection algorithm for solving the minimum-norm solution of variational inequality and fixed point problems

Limited-Memory Greedy Quasi-Newton Method with Non-asymptotic Superlinear Convergence Rate

A Limited Memory Algorithm for Bound Constrained Optimization

Convergence Analysis of Projection Method for Variational Inequalities

Novel Accelerated Cyclic Iterative Approximation for Hierarchical Variational Inequalities Constrained by Multiple-Set Split Common Fixed-Point Problems

Projection-type method with line-search process for solving variational inequalities

A Self-Adaptive Projection Method for A Class of Variant Variational Inequalities

A Projection-Based Algorithm for Solving Stochastic Inverse Variational Inequality Problems