Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

An Efficient Inexact Newton-CG Algorithm for the Smallest Enclosing Ball Problem of Large Dimensions

Approximation Algorithms for Smallest Intersecting Balls

A stochastic moving ball approximation method for smooth convex constrained minimization

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization

A dual Simplex-type algorithm for the smallest enclosing ball of balls and related problems

An Efficient Sieving-Based Secant Method for Sparse Optimization Problems with Least-Squares Constraints

An efficient inertial subspace minimization CG algorithm with convergence rate analysis for constrained nonlinear monotone equations

Smale's α-theory for inexact Newton methods under the γ-condition

A Whole New Ball Game: A Primal Accelerated Method for Matrix Games and Minimizing the Maximum of Smooth Functions

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

An efficient sieving based secant method for sparse optimization problems with least-squares constraints

Optimization Algorithms for Faster Computational Geometry.

Semismooth Newton Algorithm for Efficient Projections onto $\ell_1, ∞$-norm Ball

Smooth Distance Approximation

A Globalized Inexact Semismooth Newton Method for Nonsmooth Fixed-point Equations involving Variational Inequalities

A Slightly Lifted Convex Relaxation for Nonconvex Quadratic Programming with Ball Constraints

A Newton-CG Augmented Lagrangian Method for Semidefinite Programming

On Lower Complexity Bounds for Large-Scale Smooth Convex Optimization

A hybrid inexact regularized Newton and negative curvature method

Semismooth Newton Algorithm for Efficient Projections Onto ℓ1, ∞-Norm Ball