Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

Regularized Rényi divergence minimization through Bregman proximal gradient algorithms

Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

Interpolating between Optimal Transport and KL regularized Optimal Transport using Rényi Divergences

Stochastic Methods in Variational Inequalities: Ergodicity, Bias and Refinements

Bregman Proximal Langevin Monte Carlo via Bregman--Moreau Envelopes

Stochastic variance-reduced Gaussian variational inference on the Bures-Wasserstein manifold

Stochastic Bregman extragradient algorithm with line search for stochastic mixed variational inequalities

Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

A Deterministic Global Optimization Method for Variational Inference

Stochastic Gradient Descent as Approximate Bayesian Inference

Unbiased Kinetic Langevin Monte Carlo with Inexact Gradients

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Optimistic Dual Extrapolation for Coherent Non-monotone Variational Inequalities

Efficient Sampling on Riemannian Manifolds via Langevin MCMC

Accelerated Bregman gradient methods for relatively smooth and relatively Lipschitz continuous minimization problems

Linear convergence of proximal descent schemes on the Wasserstein space

On quasi-inertial double extragradient-type iterations for variational inequalities and Bregman relatively asymptotically nonexpansive mapping

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

Solving Fredholm Integral Equations of the First Kind via Wasserstein Gradient Flows

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization