Abstract:We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly non-differentiable but convex component. We propose a proximal stochastic gradient algorithm based on variance reduction, called ProxSVRG+. Our main contribution lies in the analysis of ProxSVRG+. It recovers several existing convergence results and improves/generalizes them (in terms of the number of stochastic gradient oracle calls and proximal oracle calls). In particular, ProxSVRG+ generalizes the best results given by the SCSG algorithm, recently proposed by [Lei et al., 2017] for the smooth nonconvex case. ProxSVRG+ is also more straightforward than SCSG and yields simpler analysis. Moreover, ProxSVRG+ outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, which partially solves an open problem proposed in [Reddi et al., 2016]. Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG [Reddi et al., 2016]. Moreover, for nonconvex functions satisfied Polyak-Lojasiewicz condition, we prove that ProxSVRG+ achieves a global linear convergence rate without restart unlike ProxSVRG. Thus, it can automatically switch to the faster linear convergence in some regions as long as the objective function satisfies the PL condition locally in these regions. Finally, we conduct several experiments and the experimental results are consistent with the theoretical results.

Proximal Subgradient Algorithm for a Class of Nonconvex Bilevel Equilibrium Problems

Iterative algorithms for equilibrium problems based on proximal-like methods

A Two-Step Proximal Point Algorithm for Nonconvex Equilibrium Problems with Applications to Fractional Programming

Proximal Algorithms for a class of abstract convex functions

Simple proximal-type algorithms for equilibrium problems

A Bregman proximal subgradient algorithm for nonconvex and nonsmooth fractional optimization problems

On the convergence of proximal gradient methods for convex simple bilevel optimization

A Proximal-Gradient Method for Constrained Optimization

Relaxed-Inertial Proximal Point Algorithms for Nonconvex Equilibrium Problems with Applications

An extragradient projection method for strongly quasiconvex equilibrium problems with applications

Proximal Point Algorithms with Inertial Extrapolation for Quasi-convex Pseudo-monotone Equilibrium Problems

Convergence of inertial prox-penalization and inertial forward–backward algorithms for solving monotone bilevel equilibrium problems

An accelerated subgradient extragradient algorithm for solving bilevel variational inequality problems involving non-Lipschitz operator

An interior proximal gradient method for nonconvex optimization

A subgradient algorithm for a class of nonlinear split feasibility problems: application to jointly constrained Nash equilibrium models

Halpern inertial subgradient extragradient algorithm for solving equilibrium problems in Banach spaces

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

A Refined Proximal Algorithm for Nonconvex Multiobjective Optimization in Hilbert Spaces

Alternating and Parallel Proximal Gradient Methods for Nonsmooth, Nonconvex Minimax: A Unified Convergence Analysis

A Proximal Gradient Method with an Explicit Line search for Multiobjective Optimization

On the Bregman-proximal iterative algorithm for the monotone inclusion problem in Banach spaces