Abstract:Second-order optimization methods, such as cubic regularized Newton methods, are known for their rapid convergence rates; nevertheless, they become impractical in high-dimensional problems due to their substantial memory requirements and computational costs. One promising approach is to execute second-order updates within a lower-dimensional subspace, giving rise to subspace second-order methods. However, the majority of existing subspace second-order methods randomly select subspaces, consequently resulting in slower convergence rates depending on the problem's dimension $d$. In this paper, we introduce a novel subspace cubic regularized Newton method that achieves a dimension-independent global convergence rate of ${O}\left(\frac{1}{mk}+\frac{1}{k^2}\right)$ for solving convex optimization problems. Here, $m$ represents the subspace dimension, which can be significantly smaller than $d$. Instead of adopting a random subspace, our primary innovation involves performing the cubic regularized Newton update within the Krylov subspace associated with the Hessian and the gradient of the objective function. This result marks the first instance of a dimension-independent convergence rate for a subspace second-order method. Furthermore, when specific spectral conditions of the Hessian are met, our method recovers the convergence rate of a full-dimensional cubic regularized Newton method. Numerical experiments show our method converges faster than existing random subspace methods, especially for high-dimensional problems.

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Multilevel Regularized Newton Methods with Fast Convergence Rates

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

A Multilevel Method for Self-Concordant Minimization

Krylov Cubic Regularized Newton: A Subspace Second-Order Method with Dimension-Free Convergence Rate

Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study

Stochastic Sub-Sampled Newton Method with Variance Reduction

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

Newton-type multilevel optimization method

Stochastic Newton Proximal Extragradient Method

Super-Universal Regularized Newton Method

Randomized subspace regularized Newton method for unconstrained non-convex optimization

SPAN: A Stochastic Projected Approximate Newton Method

Inexact Newton-type Methods for Optimisation with Nonnegativity Constraints

Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization

A General Two-Level Subspace Method for Nonlinear Optimization

Low-Rank Extragradient Methods for Scalable Semidefinite Optimization

Approximate Newton Methods and Their Local Convergence.

A Randomized Nonlinear Rescaling Method in Large-Scale Constrained Convex Optimization

Revisiting Sub-sampled Newton Methods