Abstract:We extend traditional complexity analyses of trust-region methods for unconstrained, possibly nonconvex, optimization. Whereas most complexity analyses assume uniform boundedness of the model Hessians, we work with potentially unbounded model Hessians. Boundedness is not guaranteed in practical implementations, in particular ones based on quasi-Newton updates such as PSB, BFGS and SR1. Our analysis is conducted for a family of trust-region methods that includes most known methods as special cases. We examine two regimes of Hessian growth: one bounded by a power of the number of successful iterations, and one bounded by a power of the number of iterations. This allows us to formalize and confirm the profound intuition of Powell [IMA J. Numer. Ana. 30(1):289-301,2010], who studied convergence under a special case of our assumptions, but whose proof contained complexity arguments. Specifically, for \(0 \leq p < 1\), we establish sharp \(O(\epsilon^{-2/(1-p)})\) evaluation complexity to find an \(\epsilon\)-stationary point when model Hessians are \(O(k^p)\), where \(k\) is the iteration counter. For \(p = 1\), which is the case studied by Powell, we establish a sharp \(O(\exp(c\epsilon^{-2}))\) evaluation complexity for a certain constant \(c > 0\). This is as Powell suspected and is far worse than other bounds surmised elsewhere in the literature. We establish similar bounds when model Hessians are \(O(|\mathcal{S}_k|^p)\), where \(|\mathcal{S}_k|\) is the number of iterations where the step was accepted, up to iteration \(k\). To the best of our knowledge, ours is the first work to provide complexity bounds when model Hessians grow linearly with \(|\mathcal{S}_k|\) or at most linearly with \(k\), which covers multiple quasi-Newton approximations.

First- and Second-Order High Probability Complexity Bounds for Trust-Region Methods with Noisy Oracles

High-Probability Complexity Bounds for Non-smooth Stochastic Convex Optimization with Heavy-Tailed Noise

High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

Trust Region Methods for Nonconvex Stochastic Optimization Beyond Lipschitz Smoothness

High Probability Complexity Bounds for Adaptive Step Search Based on Stochastic Oracles

A Trust-Region Algorithm for Noisy Equality Constrained Optimization

Complexity of a Class of First-Order Objective-Function-Free Optimization Algorithms

A Fully Stochastic Second-Order Trust Region Method

A non-monotone trust-region method with noisy oracles and additional sampling

Hölderian Error Bounds and Kurdyka-Łojasiewicz Inequality for the Trust Region Subproblem

Complexity of Adagrad and other first-order methods for nonconvex optimization problems with bounds constraints

Complexity of Zeroth- and First-order Stochastic Trust-Region Algorithms

A robust trust region method for nonlinear optimization with inequality constraint

High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance

Beyond Nonconvexity: A Universal Trust-Region Method with New Analyses

Minimax Excess Risk of First-Order Methods for Statistical Learning with Data-Dependent Oracles

Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models

Complexity of trust-region methods in the presence of unbounded Hessian approximations

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function