Abstract:We consider minimizing finite-sum and expectation objective functions via Hessian-averaging based subsampled Newton methods. These methods allow for gradient inexactness and have fixed per-iteration Hessian approximation costs. The recent work (Na et al. 2023) demonstrated that Hessian averaging can be utilized to achieve fast $\mathcal{O}\left(\sqrt{\tfrac{\log k}{k}}\right)$ local superlinear convergence for strongly convex functions in high probability, while maintaining fixed per-iteration Hessian costs. These methods, however, require gradient exactness and strong convexity, which poses challenges for their practical implementation. To address this concern we consider Hessian-averaged methods that allow gradient inexactness via norm condition based adaptive-sampling strategies. For the finite-sum problem we utilize deterministic sampling techniques which lead to global linear and sublinear convergence rates for strongly convex and nonconvex functions respectively. In this setting we are able to derive an improved deterministic local superlinear convergence rate of $\mathcal{O}\left(\tfrac{1}{k}\right)$. For the %expected risk expectation problem we utilize stochastic sampling techniques, and derive global linear and sublinear rates for strongly convex and nonconvex functions, as well as a $\mathcal{O}\left(\tfrac{1}{\sqrt{k}}\right)$ local superlinear convergence rate, all in expectation. We present novel analysis techniques that differ from the previous probabilistic results. Additionally, we propose scalable and efficient variations of these methods via diagonal approximations and derive the novel diagonally-averaged Newton (Dan) method for large-scale problems. Our numerical results demonstrate that the Hessian averaging not only helps with convergence, but can lead to state-of-the-art performance on difficult problems such as CIFAR100 classification with ResNets.

A Multilevel Method for Self-Concordant Minimization

A Multilevel Method for Self-Concordant Minimization

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Multilevel Regularized Newton Methods with Fast Convergence Rates

On high-order multilevel optimization strategies

Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence

Newton-type multilevel optimization method

Revisiting Sub-sampled Newton Methods

Self-concordant Smoothing for Large-Scale Convex Composite Optimization

Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization

Sketch-and-Project Meets Newton Method: Global $\mathcal O(k^{-2})$ Convergence with Low-Rank Updates

LSEMINK: A Modified Newton-Krylov Method for Log-Sum-Exp Minimization

A Unifying Framework for Convergence Analysis of Approximate Newton Methods.

Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches

Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses

A Subsampling Line-Search Method with Second-Order Results

Improved global performance guarantees of second-order methods in convex minimization

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods

A self-scaling memoryless BFGS based conjugate gradient method using multi-step secant condition for unconstrained minimization

Super-Universal Regularized Newton Method

An efficient inertial subspace minimization CG algorithm with convergence rate analysis for constrained nonlinear monotone equations