Abstract:We consider minimizing finite-sum and expectation objective functions via Hessian-averaging based subsampled Newton methods. These methods allow for gradient inexactness and have fixed per-iteration Hessian approximation costs. The recent work (Na et al. 2023) demonstrated that Hessian averaging can be utilized to achieve fast $\mathcal{O}\left(\sqrt{\tfrac{\log k}{k}}\right)$ local superlinear convergence for strongly convex functions in high probability, while maintaining fixed per-iteration Hessian costs. These methods, however, require gradient exactness and strong convexity, which poses challenges for their practical implementation. To address this concern we consider Hessian-averaged methods that allow gradient inexactness via norm condition based adaptive-sampling strategies. For the finite-sum problem we utilize deterministic sampling techniques which lead to global linear and sublinear convergence rates for strongly convex and nonconvex functions respectively. In this setting we are able to derive an improved deterministic local superlinear convergence rate of $\mathcal{O}\left(\tfrac{1}{k}\right)$. For the %expected risk expectation problem we utilize stochastic sampling techniques, and derive global linear and sublinear rates for strongly convex and nonconvex functions, as well as a $\mathcal{O}\left(\tfrac{1}{\sqrt{k}}\right)$ local superlinear convergence rate, all in expectation. We present novel analysis techniques that differ from the previous probabilistic results. Additionally, we propose scalable and efficient variations of these methods via diagonal approximations and derive the novel diagonally-averaged Newton (Dan) method for large-scale problems. Our numerical results demonstrate that the Hessian averaging not only helps with convergence, but can lead to state-of-the-art performance on difficult problems such as CIFAR100 classification with ResNets.

HOPS: high-performance library for (non-)uniform sampling of convex-constrained models

hopsy — a methods marketplace for convex polytope sampling in Python

PolytopeWalk: Sparse MCMC Sampling over Polytopes

Proximal Oracles for Optimization and Sampling

SHPOS: A Theoretical Guaranteed Accelerated Particle Optimization Sampling Method

POLYTOPE: a flexible sampling system for answering exploratory queries

OPES: On-the-fly Probability Enhanced Sampling Method

General Purpose Optimization Library (GPOL): A Flexible and Efficient Multi-Purpose Optimization Library in Python

modOpt: A modular development environment and library for optimization algorithms

HOPS: A Fast Algorithm for Segmenting Piecewise Polynomials of Arbitrary Orders

An adjoint-free algorithm for conditional nonlinear optimal perturbations (CNOPs) via sampling

MesoHops: Size-invariant scaling calculations of multi-excitation open quantum systems

High-performance sampling of generic Determinantal Point Processes

Polytope Division Method: A Scalable Sampling Method for Problems with High-dimensional Parameters

gapsplit: Efficient random sampling for non-convex constraint-based models

HomOpt: A Homotopy-Based Hyperparameter Optimization Method

Auptimizer -- an Extensible, Open-Source Framework for Hyperparameter Tuning

A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences

Fast Unconstrained Optimization via Hessian Averaging and Adaptive Gradient Sampling Methods

Algorithm xxxx: HiPPIS A High-Order Positivity-Preserving Mapping Software for Structured Meshes

POAS: a framework for exploiting accelerator level parallelism in heterogeneous environments