Abstract:We consider minimizing finite-sum and expectation objective functions via Hessian-averaging based subsampled Newton methods. These methods allow for gradient inexactness and have fixed per-iteration Hessian approximation costs. The recent work (Na et al. 2023) demonstrated that Hessian averaging can be utilized to achieve fast $\mathcal{O}\left(\sqrt{\tfrac{\log k}{k}}\right)$ local superlinear convergence for strongly convex functions in high probability, while maintaining fixed per-iteration Hessian costs. These methods, however, require gradient exactness and strong convexity, which poses challenges for their practical implementation. To address this concern we consider Hessian-averaged methods that allow gradient inexactness via norm condition based adaptive-sampling strategies. For the finite-sum problem we utilize deterministic sampling techniques which lead to global linear and sublinear convergence rates for strongly convex and nonconvex functions respectively. In this setting we are able to derive an improved deterministic local superlinear convergence rate of $\mathcal{O}\left(\tfrac{1}{k}\right)$. For the %expected risk expectation problem we utilize stochastic sampling techniques, and derive global linear and sublinear rates for strongly convex and nonconvex functions, as well as a $\mathcal{O}\left(\tfrac{1}{\sqrt{k}}\right)$ local superlinear convergence rate, all in expectation. We present novel analysis techniques that differ from the previous probabilistic results. Additionally, we propose scalable and efficient variations of these methods via diagonal approximations and derive the novel diagonally-averaged Newton (Dan) method for large-scale problems. Our numerical results demonstrate that the Hessian averaging not only helps with convergence, but can lead to state-of-the-art performance on difficult problems such as CIFAR100 classification with ResNets.

Convergence Analysis of the Fast Subspace Descent Methods for Convex Optimization Problems

Convergence analysis of the Fast Subspace Descent method for convex optimization problems

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

On Some Versions of Subspace Optimization Methods with Inexact Gradient Information

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

A full approximation scheme multilevel method for nonlinear variational inequalities

Fast convex optimization via closed-loop time scaling of gradient dynamics

Convergence rate analysis of the gradient descent–ascent method for convex–concave saddle-point problems

A Unifying Framework for Convergence Analysis of Approximate Newton Methods.

A new unified framework for designing convex optimization methods with prescribed theoretical convergence estimates: A numerical analysis approach

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

A gradient descent akin method for constrained optimization: algorithms and applications

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods

Linearly-Convergent FISTA Variant for Composite Optimization with Duality

Descent Methods for Vector Optimization Problems: A Majorization-Minimization Perspective

Gradient descent with nonconvex constraints: local concavity determines convergence

Fast-convergent two-level restricted additive Schwarz methods based on optimal local approximation spaces