Abstract:We consider minimizing finite-sum and expectation objective functions via Hessian-averaging based subsampled Newton methods. These methods allow for gradient inexactness and have fixed per-iteration Hessian approximation costs. The recent work (Na et al. 2023) demonstrated that Hessian averaging can be utilized to achieve fast $\mathcal{O}\left(\sqrt{\tfrac{\log k}{k}}\right)$ local superlinear convergence for strongly convex functions in high probability, while maintaining fixed per-iteration Hessian costs. These methods, however, require gradient exactness and strong convexity, which poses challenges for their practical implementation. To address this concern we consider Hessian-averaged methods that allow gradient inexactness via norm condition based adaptive-sampling strategies. For the finite-sum problem we utilize deterministic sampling techniques which lead to global linear and sublinear convergence rates for strongly convex and nonconvex functions respectively. In this setting we are able to derive an improved deterministic local superlinear convergence rate of $\mathcal{O}\left(\tfrac{1}{k}\right)$. For the %expected risk expectation problem we utilize stochastic sampling techniques, and derive global linear and sublinear rates for strongly convex and nonconvex functions, as well as a $\mathcal{O}\left(\tfrac{1}{\sqrt{k}}\right)$ local superlinear convergence rate, all in expectation. We present novel analysis techniques that differ from the previous probabilistic results. Additionally, we propose scalable and efficient variations of these methods via diagonal approximations and derive the novel diagonally-averaged Newton (Dan) method for large-scale problems. Our numerical results demonstrate that the Hessian averaging not only helps with convergence, but can lead to state-of-the-art performance on difficult problems such as CIFAR100 classification with ResNets.

Extremely Fast Convergence Rates for Extremum Seeking Control with Polyak-Ruppert Averaging

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Accelerated Almost-Sure Convergence Rates for Nonconvex Stochastic Gradient Descent using Stochastic Learning Rates

Markovian Foundations for Quasi-Stochastic Approximation with Applications to Extremum Seeking Control

Accelerated stochastic approximation with state-dependent noise

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weight

Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

Fast convergence of sample-average approximation for saddle-point problems

Exact Convergence rate of the subgradient method by using Polyak step size

The Speed-Robustness Trade-Off for First-Order Methods with Additive Gradient Noise

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate.

On Almost Sure Convergence Rates of Stochastic Gradient Methods.

Linear convergence of decentralized estimation for statistical estimation using gradient method

On Almost Sure Convergence Rates of Stochastic Gradient Methods

Stochastic Averaging in Continuous Time and Its Applications to Extremum Seeking

The stochastic Ravine accelerated gradient method with general extrapolation coefficients

Aiming towards the minimizers: fast convergence of SGD for overparametrized problems

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation