Abstract:An Adaptive Regularisation framework using Cubics (ARC) was proposed for unconstrained optimization and analysed in Cartis, Gould and Toint (Part I, Math Program, doi:10.1007/s10107-009-0286-5, 2009), generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser, Deuflhard and Erdmann (Optim Methods Softw 22(3):413–431, 2007). In this companion paper, we further the analysis by providing worst-case global iteration complexity bounds for ARC and a second-order variant to achieve approximate first-order, and for the latter second-order, criticality of the iterates. In particular, the second-order ARC algorithm requires at most \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathcal{O}(\epsilon^{-3/2})}$$\end{document} iterations, or equivalently, function- and gradient-evaluations, to drive the norm of the gradient of the objective below the desired accuracy \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\epsilon}$$\end{document}, and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathcal{O}(\epsilon^{-3})}$$\end{document} iterations, to reach approximate nonnegative curvature in a subspace. The orders of these bounds match those proved for Algorithm 3.3 of Nesterov and Polyak which minimizes the cubic model globally on each iteration. Our approach is more general in that it allows the cubic model to be solved only approximately and may employ approximate Hessians.

Examples of slow convergence for adaptive regularization optimization methods are not isolated

Fast convergence to non-isolated minima: four equivalent conditions for $\mathrm{C}^2$ functions

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Convergence rates for critical point regularization

Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity

Convergence of successive linear programming algorithms for noisy functions

On the convergence of an inertial proximal algorithm with a Tikhonov regularization term

Gradient descent with nonconvex constraints: local concavity determines convergence

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

A Stochastic Objective-Function-Free Adaptive Regularization Method with Optimal Complexity

Revisiting $L_q(0\leq q<1)$ Norm Regularized Optimization

A note on $L^1$-Convergence of the Empiric Minimizer for unbounded functions with fast growth

Linear convergence of first order methods for non-strongly convex optimization

Hölder Gradient Descent and Adaptive Regularization Methods in Banach Spaces for First-Order Points

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity

A Nesterov type algorithm with double Tikhonov regularization: fast convergence of the function values and strong convergence to the minimal norm solution

Linear Convergence of Adaptive Stochastic Gradient Descent

Quantitative Convergence of Quadratically Regularized Linear Programs