Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles

Design and Implementation of Large-scale Nonlinear Optimization Software UniOptima

Two Modified Single-Parameter Scaling Broyden–Fletcher–Goldfarb–Shanno Algorithms for Solving Nonlinear System of Symmetric Equations

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

A new inexact SQP algorithm for nonlinear systems of mixed equalities and inequalities

A hybrid algorithm for nonlinear minimax problems

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

Superlinear Optimization Algorithms

A Proximal-Gradient Method for Constrained Optimization

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobians

A subspace SQP method for equality constrained optimization

A globally convergent SQP-type method with least constraint violation for nonlinear semidefinite programming

S-SOS: Stochastic Sum-Of-Squares for Parametric Polynomial Optimization

A nonlinear relaxation-strategy-based algorithm for solving sum-of-linear-ratios problems

Black-box Optimization Algorithms for Regularized Least-squares Problems