Abstract:Update formulas for the Hessian approximations in quasi-Newton methods such as BFGS can be derived as analytical solutions to certain nearest-matrix problems. In this article, we propose a similar idea for deriving new limited memory versions of quasi-Newton methods. Most limited memory quasi-Newton methods make use of Hessian approximations that can be written as a scaled identity matrix plus a symmetric matrix with limited rank. We derive a way of finding the nearest matrix of this type to an arbitrary symmetric matrix, in either the Frobenius norm, the induced $l^2$ norm, or a dissimilarity measure for positive definite matrices in terms of trace and determinant. In doing so, we lay down a framework for more general matrix optimization problems with unitarily invariant matrix norms and arbitrary constraints on the set of eigenvalues. We then propose a trust region method in which the Hessian approximation, after having been updated by a Broyden class formula and used to solve a trust-region problem, is replaced by one of its closest limited memory approximations. We propose to store the Hessian approximation in terms of its eigenvectors and eigenvalues in a way that completely defines its eigenvalue decomposition, as this simplifies both the solution of the trust region subproblem and the nearest limited memory matrix problem. Our method is compared to a reference trust region method with the usual limited memory BFGS updates, and is shown to require fewer iterations and the storage of fewer vectors for a variety of test problems.

An $LDL^T$ Trust-Region Quasi-Newton Method

A Trust Region-Type Method for Solving Monotone Variational Inequality

A simple and practical adaptive trust-region method

Quasi-Newton Trust Region Algorithm for Non-Smooth Least Squares Problems

A Trust Region Method for Optimization Problem with Singular Solutions

A NESTED LANCZOS METHOD FOR THE TRUST-REGION SUBPROBLEM

Non-monotone Nonlinear Conjugate Gradient Method Based on the Trust Region Technique

On the Generalized Lanczos Trust-Region Method.

A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimization

Regularization of Limited Memory Quasi-Newton Methods for Large-Scale Nonconvex Minimization

A New Simple Model Trust-Region Method with Generalized Barzilai-Borwein Parameter for Large-Scale Optimization

A Fractional Trust Region Method for Linear Equality Constrained Optimization

Shape-changing trust-region methods using multipoint symmetric secant matrices

A Trust Region Algorithm with Conjugate Gradient Technique for Optimization Problems

Novel Limited Memory Quasi-Newton Methods Based On Optimal Matrix Approximation

An improved nonmonotone adaptive trust region method

An augmented Lagrangian trust-region method with inexact gradient evaluations to accelerate constrained optimization problems using model hyperreduction

Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximations

New Trust Region Method for Line Search

A Trust Region Method with Regularized Barzilai-Borwein Step-Size for Large-Scale Unconstrained Optimization

A nonmonotone trust region method for unconstrained optimization