Abstract:An Adaptive Regularisation framework using Cubics (ARC) was proposed for unconstrained optimization and analysed in Cartis, Gould and Toint (Part I, Math Program, doi:10.1007/s10107-009-0286-5, 2009), generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser, Deuflhard and Erdmann (Optim Methods Softw 22(3):413–431, 2007). In this companion paper, we further the analysis by providing worst-case global iteration complexity bounds for ARC and a second-order variant to achieve approximate first-order, and for the latter second-order, criticality of the iterates. In particular, the second-order ARC algorithm requires at most \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathcal{O}(\epsilon^{-3/2})}$$\end{document} iterations, or equivalently, function- and gradient-evaluations, to drive the norm of the gradient of the objective below the desired accuracy \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\epsilon}$$\end{document}, and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathcal{O}(\epsilon^{-3})}$$\end{document} iterations, to reach approximate nonnegative curvature in a subspace. The orders of these bounds match those proved for Algorithm 3.3 of Nesterov and Polyak which minimizes the cubic model globally on each iteration. Our approach is more general in that it allows the cubic model to be solved only approximately and may employ approximate Hessians.

Accelerating Adaptive Cubic Regularization of Newton's Method Via Random Sampling

Adaptively Accelerating Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling

On Adaptive Cubic Regularized Newton's Methods for Convex Optimization via Random Sampling

A Unified Scheme to Accelerate Adaptive Cubic Regularization and Gradient Methods for Convex Optimization

Cubic regularized subspace Newton for non-convex optimization

A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization

Solving the Cubic Regularization Model by a Nested Restarting Lanczos Method

Regularized methods via cubic model subspace minimization for nonconvex optimization

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

Stochastic Variance-Reduced Cubic Regularized Newton Method

Efficient Hyper-parameter Optimization with Cubic Regularization.

Stochastic cubic regularization for fast nonconvex optimization

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization

Adaptive Quadratically Regularized Newton Method for Riemannian Optimization

Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity

AdaCN: An Adaptive Cubic Newton Method for Nonconvex Stochastic Optimization

Subsampled cubic regularization method for finite-sum minimization

Inexact Adaptive Cubic Regularization Algorithms on Riemannian Manifolds and Application

Super-Universal Regularized Newton Method

A Unified Adaptive Tensor Approximation Scheme to Accelerate Composite Convex Optimization.

A modified adaptive cubic regularization method for large-scale unconstrained optimization problem