Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

Smoothing randomized block-coordinate proximal gradient algorithms for nonsmooth nonconvex composite optimization

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

Approximate Bregman Proximal Gradient Algorithm for Relatively Smooth Nonconvex Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

The Inexact Cyclic Block Proximal Gradient Method and Properties of Inexact Proximal Maps

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

A Decentralized Proximal Gradient Tracking Algorithm for Composite Optimization on Riemannian Manifolds

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

A Bregman proximal subgradient algorithm for nonconvex and nonsmooth fractional optimization problems

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Global convergence of block proximal iteratively reweighted algorithm with extrapolation

Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds

Barzilai-Borwein Proximal Gradient Methods for Multiobjective Composite Optimization Problems with Improved Linear Convergence

Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization

A stochastic two-step inertial Bregman proximal alternating linearized minimization algorithm for nonconvex and nonsmooth problems