Abstract:Second-order methods can address the shortcomings of first-order methods for the optimization of large-scale machine learning models. However, second-order methods have significantly higher computational costs associated with the computation of second-order information. Subspace methods that are based on randomization have addressed some of these computational costs as they compute search directions in lower dimensions. Even though super-linear convergence rates have been empirically observed, it has not been possible to rigorously show that these variants of second-order methods can indeed achieve such fast rates. Also, it is not clear whether subspace methods can be applied to non-convex cases. To address these shortcomings, we develop a link between multigrid optimization methods and low-rank Newton methods that enables us to prove the super-linear rates of stochastic low-rank Newton methods rigorously. Our method does not require any computations in the original model dimension. We further propose a truncated version of the method that is capable of solving high-dimensional non-convex problems. Preliminary numerical experiments show that our method has a better escape rate from saddle points compared to accelerated gradient descent and Adam and thus returns lower training errors.

Newton-type multilevel optimization method

Multilevel Regularized Newton Methods with Fast Convergence Rates

On high-order multilevel optimization strategies

Structural Topology Optimization Using a Level Set Method with Finite Difference Updating Scheme

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

A Multilevel Method for Self-Concordant Minimization

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Semismooth Newton-type method for bilevel optimization: Global convergence and extensive numerical experiments

A Line Search Multigrid Method for Large-Scale Nonlinear Optimization

Towards Differentiable Multilevel Optimization: A Gradient-Based Approach

A Line Search Multigrid Method for Large-Scale Convex Optimization

Multilevel Geometric Optimization for Regularised Constrained Linear Inverse Problems

Multigrid method for nonlinear eigenvalue problems based on Newton iteration

Scalable Multilevel Monte Carlo Methods Exploiting Parallel Redistribution on Coarse Levels

Coderivative-Based Newton Methods in Structured Nonconvex and Nonsmooth Optimization

A Unifying Framework for Convergence Analysis of Approximate Newton Methods.

Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization

A multiscale Consensus-Based algorithm for multi-level optimization

On the Convergence of Newton-type Proximal Gradient Method for Multiobjective Optimization Problems

Multilevel (hierarchical) Optimization: Complexity Issues, Optimality Conditions, Algorithms