Abstract:In many contemporary optimization problems such as those arising in machine learning, it can be computationally challenging or even infeasible to evaluate an entire function or its derivatives. This motivates the use of stochastic algorithms that sample problem data, which can jeopardize the guarantees obtained through classical globalization techniques in optimization, such as a line search. Using subsampled function values is particularly challenging for the latter strategy, which relies upon multiple evaluations. For nonconvex data-related problems, such as training deep learning models, one aims at developing methods that converge to second-order stationary points quickly, that is, escape saddle points efficiently. This is particularly difficult to ensure when one only accesses subsampled approximations of the objective and its derivatives. In this paper, we describe a stochastic algorithm based on negative curvature and Newton-type directions that are computed for a subsampling model of the objective. A line-search technique is used to enforce suitable decrease for this model; for a sufficiently large sample, a similar amount of reduction holds for the true objective. We then present worst-case complexity guarantees for a notion of stationarity tailored to the subsampling context. Our analysis encompasses the deterministic regime and allows us to identify sampling requirements for second-order line-search paradigms. As we illustrate through real data experiments, these worst-case estimates need not be satisfied for our method to be competitive with first-order strategies in practice.

Stochastic Sub-Sampled Newton Method with Variance Reduction

Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches

Accelerated Stochastic ADMM with Variance Reduction

Stochastic Nested Variance Reduction for Nonconvex Optimization

Stochastic Zeroth-order Optimization Via Variance Reduction Method.

Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference.

VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning: Part I

Adaptive Variance Reducing for Stochastic Gradient Descent.

Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning: Part II

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction

Stochastic Variance-Reduced Cubic Regularized Newton Method

Revisiting Sub-sampled Newton Methods

Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradients

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

A Subsampling Line-Search Method with Second-Order Results

Variance reduction techniques for stochastic proximal point algorithms

Stochastic variance reduced gradient method for linear ill-posed inverse problems

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness

A stochastic gradient method with variance control and variable learning rate for Deep Learning