Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

Quadratically Regularized Subgradient Methods for Weakly Convex Optimization with Weakly Convex Constraints

First-Order Methods for Nonsmooth Nonconvex Functional Constrained Optimization with or without Slater Points

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning

Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization

A Sequential Quadratic Programming Algorithm for Nonconvex, Nonsmooth Constrained Optimization

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

A quadratically convergent semismooth Newton method for nonlinear semidefinite programming without generalized Jacobian regularity

Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization

Revisiting $L_q(0\leq q<1)$ Norm Regularized Optimization

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Proximal-Gradient Method for Constrained Optimization

Avoiding strict saddle points of nonconvex regularized problems

Cubic regularized subspace Newton for non-convex optimization

Nonconvex Sparse Logistic Regression with Weakly Convex Regularization

Oracle Complexity of Single-Loop Switching Subgradient Methods for Non-Smooth Weakly Convex Functional Constrained Optimization

Smoothing SQP methods for solving degenerate nonsmooth constrained optimization problems with applications to bilevel programs

Accelerated Stochastic Optimization Methods under Quasar-convexity

A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles

Stochastic Weakly Convex Optimization Beyond Lipschitz Continuity