Abstract:We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly non-differentiable but convex component. We propose a proximal stochastic gradient algorithm based on variance reduction, called ProxSVRG+. Our main contribution lies in the analysis of ProxSVRG+. It recovers several existing convergence results and improves/generalizes them (in terms of the number of stochastic gradient oracle calls and proximal oracle calls). In particular, ProxSVRG+ generalizes the best results given by the SCSG algorithm, recently proposed by [Lei et al., 2017] for the smooth nonconvex case. ProxSVRG+ is also more straightforward than SCSG and yields simpler analysis. Moreover, ProxSVRG+ outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, which partially solves an open problem proposed in [Reddi et al., 2016]. Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG [Reddi et al., 2016]. Moreover, for nonconvex functions satisfied Polyak-Lojasiewicz condition, we prove that ProxSVRG+ achieves a global linear convergence rate without restart unlike ProxSVRG. Thus, it can automatically switch to the faster linear convergence in some regions as long as the objective function satisfies the PL condition locally in these regions. Finally, we conduct several experiments and the experimental results are consistent with the theoretical results.

On Proximal Gradient Method for the Convex Problems Regularized with the Group Reproducing Kernel Norm

On the Linear Convergence of a Proximal Gradient Method for a Class of Nonsmooth Convex Minimization Problems

On the Proximal Landweber Newton Method for a Class of Nonsmooth Convex Problems.

A Modified Proximal Gradient Method for a Family of Nonsmooth Convex Optimization Problems

A proximal-gradient method for problems with overlapping group-sparse regularization: support identification complexity

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

A Modified Proximal Gradient Method for a Class of Sparse Optimization Problems

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Proximal-Gradient Method for Constrained Optimization

A Smoothing Proximal Gradient Algorithm for Nonsmooth Convex Regression with Cardinality Penalty.

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Efficient Inexact Proximal Gradient Algorithms for Structured Sparsity-Inducing Norm.

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Fast Learning of Nonconvex `1-2-Regularizer using the Proximal Gradient Algorithm

A Proximal Iteratively Regularized Gauss–Newton Method for Nonlinear Inverse Problems

Proximal Gradient Method with Automatic Selection of the Parameter by Automatic Differentiation.

Inexact Proximal Gradient Methods for Non-convex and Non-smooth Optimization

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

A linearly convergent method for solving high-order proximal operator

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization