Abstract:High-order tensor methods for solving both convex and nonconvex optimization problems have generated significant research interest, leading to algorithms with optimal global rates of convergence and local rates that are faster than Newton's method. On each iteration, these methods require the unconstrained local minimization of a (potentially nonconvex) multivariate polynomial of degree higher than two, constructed using third-order (or higher) derivative information, and regularized by an appropriate power of regularization. Developing efficient techniques for solving such subproblems is an ongoing topic of research, and this paper addresses the case of the third-order tensor subproblem. We propose the CQR algorithmic framework, for minimizing a nonconvex Cubic multivariate polynomial with Quartic Regularisation, by minimizing a sequence of local quadratic models that incorporate simple cubic and quartic terms. The role of the cubic term is to crudely approximate local tensor information, while the quartic one controls model regularization and progress. We provide necessary and sufficient optimality conditions that fully characterise the global minimizers of these cubic-quartic models. We then turn these conditions into secular equations that can be solved using nonlinear eigenvalue techniques. We show, using our optimality characterisations, that a CQR algorithmic variant has the optimal-order evaluation complexity of $\mathcal{O}(\epsilon^{-3/2})$ when applied to minimizing our quartically-regularised cubic subproblem, which can be further improved in special cases. We propose practical CQR variants that use local tensor information to construct the local cubic-quartic models. We test these variants numerically and observe them to be competitive with ARC and other subproblem solvers on typical instances and even superior on ill-conditioned subproblems with special structure.

Implementable tensor methods in unconstrained convex optimization

Higher-order tensor methods for minimizing difference of convex functions

Near-optimal tensor methods for minimizing the gradient norm of convex functions and accelerated primal–dual tensor methods

A Sub-sampled Tensor Method for Non-convex Optimization

Near-optimal tensor methods for minimizing the gradient norm of convex functions and accelerated primal-dual tensor methods

On high-order multilevel optimization strategies

Optimal Calculation of Tensor Learning Approaches.

Properties and methods for finding the best rank-one approximation to higher-order tensors

Global Convergence of High-Order Regularization Methods with Sums-of-Squares Taylor Models

OPTAMI: Global Superlinear Convergence of High-order Methods

Moving higher-order Taylor approximations method for smooth constrained minimization problems

Efficiency of First-Order Methods for Low-Rank Tensor Recovery with the Tensor Nuclear Norm Under Strict Complementarity

Random Gradient-Free Minimization of Convex Functions

Local and Global Convergence of General Burer-Monteiro Tensor Optimizations

Cubic-quartic regularization models for solving polynomial subproblems in third-order tensor methods

Towards Unified Acceleration of High-Order Algorithms under H\"{o}lder Continuity and Uniform Convexity

Combined Methods for Solving Degenerate Unconstrained Optimization Problems

New Approach to Global Minimization of Normal Multivariate Polynomial Based on Tensor

Accelerated First-Order Optimization under Nonlinear Constraints

First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problems

Tensor Completion via Integer Optimization