Abstract:An Adaptive Regularisation framework using Cubics (ARC) was proposed for unconstrained optimization and analysed in Cartis, Gould and Toint (Part I, Math Program, doi:10.1007/s10107-009-0286-5, 2009), generalizing at the same time an unpublished method due to Griewank (Technical Report NA/12, 1981, DAMTP, University of Cambridge), an algorithm by Nesterov and Polyak (Math Program 108(1):177–205, 2006) and a proposal by Weiser, Deuflhard and Erdmann (Optim Methods Softw 22(3):413–431, 2007). In this companion paper, we further the analysis by providing worst-case global iteration complexity bounds for ARC and a second-order variant to achieve approximate first-order, and for the latter second-order, criticality of the iterates. In particular, the second-order ARC algorithm requires at most \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathcal{O}(\epsilon^{-3/2})}$$\end{document} iterations, or equivalently, function- and gradient-evaluations, to drive the norm of the gradient of the objective below the desired accuracy \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\epsilon}$$\end{document}, and \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathcal{O}(\epsilon^{-3})}$$\end{document} iterations, to reach approximate nonnegative curvature in a subspace. The orders of these bounds match those proved for Algorithm 3.3 of Nesterov and Polyak which minimizes the cubic model globally on each iteration. Our approach is more general in that it allows the cubic model to be solved only approximately and may employ approximate Hessians.

Lower Bounds for Higher-Order Convex Optimization

On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

An optimal lower bound for smooth convex functions

On Lower Complexity Bounds for Large-Scale Smooth Convex Optimization

Lower and Upper Bounds on the Generalization of Stochastic Exponentially Concave Optimization.

On high-order multilevel optimization strategies

Tight Lower Bounds under Asymmetric High-Order Hölder Smoothness and Uniform Convexity

Complexity of Minimizing Regularized Convex Quadratic Functions

Curvature and complexity: Better lower bounds for geodesically convex optimization

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Convex and Non-convex Optimization Under Generalized Smoothness

Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity

Higher-order tensor methods for minimizing difference of convex functions

Tight Lower Complexity Bounds for Strongly Convex Finite-Sum Optimization

A General Analysis Framework of Lower Complexity Bounds for Finite-Sum Optimization

Worst-case evaluation complexity and optimality of second-order methods for nonconvex smooth optimization

Non-Euclidean High-Order Smooth Convex Optimization

On Linear Convergence in Smooth Convex-Concave Bilinearly-Coupled Saddle-Point Optimization: Lower Bounds and Optimal Algorithms

Information Complexity of Mixed-integer Convex Optimization

Advancing the lower bounds: An accelerated, stochastic, second-order method with optimal adaptation to inexactness

Improved Algorithms for Convex-Concave Minimax Optimization