Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

A Proximal-Proximal Majorization-Minimization Algorithm for Nonconvex Rank Regression Problems.

A Sparse Semismooth Newton Based Proximal Majorization-Minimization Algorithm for Nonconvex Square-Root-loss Regression Problems

Proximal Dogleg Opportunistic Majorization for Nonconvex and Nonsmooth Optimization

Proximal Projection Method for Stable Linearly Constrained Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

ProPPA: A Fast Algorithm for $\ell_1$ Minimization and Low-Rank Matrix Completion

Relaxed Majorization-Minimization for Non-Smooth and Non-Convex Optimization

A Proximal-Gradient Method for Constrained Optimization

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

A Feasible Level Proximal Point Method for Nonconvex Sparse Constrained Optimization

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Generalized Majorization-Minimization for Non-Convex Optimization.

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A modified inertial proximal minimization algorithm for structured nonconvex and nonsmooth problem

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Fast Low-Rank Matrix Learning with Nonconvex Regularization

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds