Abstract:Second-order optimization methods, such as cubic regularized Newton methods, are known for their rapid convergence rates; nevertheless, they become impractical in high-dimensional problems due to their substantial memory requirements and computational costs. One promising approach is to execute second-order updates within a lower-dimensional subspace, giving rise to subspace second-order methods. However, the majority of existing subspace second-order methods randomly select subspaces, consequently resulting in slower convergence rates depending on the problem's dimension $d$. In this paper, we introduce a novel subspace cubic regularized Newton method that achieves a dimension-independent global convergence rate of ${O}\left(\frac{1}{mk}+\frac{1}{k^2}\right)$ for solving convex optimization problems. Here, $m$ represents the subspace dimension, which can be significantly smaller than $d$. Instead of adopting a random subspace, our primary innovation involves performing the cubic regularized Newton update within the Krylov subspace associated with the Hessian and the gradient of the objective function. This result marks the first instance of a dimension-independent convergence rate for a subspace second-order method. Furthermore, when specific spectral conditions of the Hessian are met, our method recovers the convergence rate of a full-dimensional cubic regularized Newton method. Numerical experiments show our method converges faster than existing random subspace methods, especially for high-dimensional problems.

Convergence analysis of a regularized Newton method with generalized regularization terms for convex optimization problems

Generalized Newton Method with Positive Definite Regularization for Nonsmooth Optimization Problems with Nonisolated Solutions

Super-Universal Regularized Newton Method

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Cubic regularized subspace Newton for non-convex optimization

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

Generalized Metric Subregularity with Applications to High-Order Regularized Newton Methods

A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization

Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses

Convergence Analysis of an Adaptively Regularized Natural Gradient Method

Multilevel Regularized Newton Methods with Fast Convergence Rates

Adaptive Quadratically Regularized Newton Method for Riemannian Optimization

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

Quantitative Convergence of Quadratically Regularized Linear Programs

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

Convex and Non-convex Optimization Under Generalized Smoothness

Randomized subspace regularized Newton method for unconstrained non-convex optimization

Global non-asymptotic super-linear convergence rates of regularized proximal quasi-Newton methods on non-smooth composite problems

A hybrid inexact regularized Newton and negative curvature method