Abstract:Stochastic gradient algorithms estimate the gradient based on only one or a few samples and enjoy low computational cost per iteration. They have been widely used in large-scale optimization problems. However, stochastic gradient algorithms are usually slow to converge and achieve sub-linear convergence rates, due to the inherent variance in the gradient computation. To accelerate the convergence, some variance-reduced stochastic gradient algorithms, e.g., proximal stochastic variance-reduced gradient (Prox-SVRG) algorithm, have recently been proposed to solve strongly convex problems. Under the strongly convex condition, these variance-reduced stochastic gradient algorithms achieve a linear convergence rate. However, many machine learning problems are convex but not strongly convex. In this paper, we introduce Prox-SVRG and its projected variant called Variance-Reduced Projected Stochastic Gradient (VRPSG) to solve a class of non-strongly convex optimization problems widely used in machine learning. As the main technical contribution of this paper, we show that both VRPSG and Prox-SVRG achieve a linear convergence rate without strong convexity. A key ingredient in our proof is a Semi-Strongly Convex (SSC) inequality which is the first to be rigorously proved for a class of non-strongly convex problems in both constrained and regularized settings. Moreover, the SSC inequality is independent of algorithms and may be applied to analyze other stochastic gradient algorithms besides VRPSG and Prox-SVRG, which may be of independent interest. To the best of our knowledge, this is the first work that establishes the linear convergence rate for the variance-reduced stochastic gradient algorithms on solving both constrained and regularized problems without strong convexity.

SPIRAL: a superlinearly convergent incremental proximal algorithm for nonconvex finite sum minimization

SPIRAL: A superlinearly convergent incremental proximal algorithm for nonconvex finite sum minimization

Proximal-Like Incremental Aggregated Gradient Method with Linear Convergence under Bregman Distance Growth Conditions

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

On the convergence of the iterates of proximal gradient algorithm with extrapolation for convex nonsmooth minimization problems

SPRING: A Fast Stochastic Proximal Alternating Method for Non-Smooth Non-Convex Optimization

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

Smooth over-parameterized solvers for non-smooth structured optimization

An Alternating Proximal Splitting Method With Global Convergence For Nonconvex Structured Sparsity Optimization

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

The Spiral Optimization Algorithm: Convergence Conditions and Settings

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

Local Linear Convergence of Forward-Backward under Partial Smoothness

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Linear Convergence of Variance-Reduced Stochastic Gradient without Strong Convexity