Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

Variance reduction techniques for stochastic proximal point algorithms

New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

A Unified Theory of Stochastic Proximal Point Methods without Smoothness

Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

Stochastic Sub-Sampled Newton Method with Variance Reduction

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

Sharper Analysis for Minibatch Stochastic Proximal Point Methods: Stability, Smoothness, and Deviation.

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Proximal Stochastic Quasi-Newton Algorithm with Dynamical Sampling and Stochastic Line Search

The stochastic proximal distance algorithm

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization.

Stochastic alternating structure-adapted proximal gradient descent method with variance reduction for nonconvex nonsmooth optimization

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization

Stochastic Newton Proximal Extragradient Method

Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches