Abstract:We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly non-differentiable but convex component. We propose a proximal stochastic gradient algorithm based on variance reduction, called ProxSVRG+. Our main contribution lies in the analysis of ProxSVRG+. It recovers several existing convergence results and improves/generalizes them (in terms of the number of stochastic gradient oracle calls and proximal oracle calls). In particular, ProxSVRG+ generalizes the best results given by the SCSG algorithm, recently proposed by [Lei et al., 2017] for the smooth nonconvex case. ProxSVRG+ is also more straightforward than SCSG and yields simpler analysis. Moreover, ProxSVRG+ outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, which partially solves an open problem proposed in [Reddi et al., 2016]. Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG [Reddi et al., 2016]. Moreover, for nonconvex functions satisfied Polyak-Lojasiewicz condition, we prove that ProxSVRG+ achieves a global linear convergence rate without restart unlike ProxSVRG. Thus, it can automatically switch to the faster linear convergence in some regions as long as the objective function satisfies the PL condition locally in these regions. Finally, we conduct several experiments and the experimental results are consistent with the theoretical results.

Smoothing Accelerated Proximal Gradient Method with Fast Convergence Rate for Nonsmooth Convex Optimization Beyond Differentiability

Smoothing Accelerated Proximal Gradient Method with Fast Convergence Rate for Nonsmooth Multi-objective Optimization

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

More Efficient Accelerated Proximal Algorithm for Nonconvex Problems

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Accelerated Proximal Gradient Methods for Nonconvex Programming

Accelerated Proximal Coordinate Gradient Methods

A feasible smoothing accelerated projected gradient method for nonsmooth convex optimization

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

Efficient Inexact Proximal Gradient Algorithm for Nonconvex Problems.

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems