Abstract:Stochastic gradient algorithms estimate the gradient based on only one or a few samples and enjoy low computational cost per iteration. They have been widely used in large-scale optimization problems. However, stochastic gradient algorithms are usually slow to converge and achieve sub-linear convergence rates, due to the inherent variance in the gradient computation. To accelerate the convergence, some variance-reduced stochastic gradient algorithms, e.g., proximal stochastic variance-reduced gradient (Prox-SVRG) algorithm, have recently been proposed to solve strongly convex problems. Under the strongly convex condition, these variance-reduced stochastic gradient algorithms achieve a linear convergence rate. However, many machine learning problems are convex but not strongly convex. In this paper, we introduce Prox-SVRG and its projected variant called Variance-Reduced Projected Stochastic Gradient (VRPSG) to solve a class of non-strongly convex optimization problems widely used in machine learning. As the main technical contribution of this paper, we show that both VRPSG and Prox-SVRG achieve a linear convergence rate without strong convexity. A key ingredient in our proof is a Semi-Strongly Convex (SSC) inequality which is the first to be rigorously proved for a class of non-strongly convex problems in both constrained and regularized settings. Moreover, the SSC inequality is independent of algorithms and may be applied to analyze other stochastic gradient algorithms besides VRPSG and Prox-SVRG, which may be of independent interest. To the best of our knowledge, this is the first work that establishes the linear convergence rate for the variance-reduced stochastic gradient algorithms on solving both constrained and regularized problems without strong convexity.

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Proximal Point Algorithms on Hadamard Manifolds: Linear Convergence and Finite Termination

Convergence rate of inexact proximal point algorithms for operator with holder metric subregularity

Convergence analysis of inexact proximal point algorithms on Hadamard manifolds

Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

Inexact FPPA for the $\ell_0$ Sparse Regularization Problem

Plug‐and‐play algorithms for convex non‐convex regularization: Convergence analysis and applications

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

Revisiting $L_q(0\leq q<1)$ Norm Regularized Optimization

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Generalized Convergence Rates Results for Linear Inverse Problems in Hilbert Spaces

Inexact Proximal Gradient Methods for Non-convex and Non-smooth Optimization

Group sparse optimization via l

Linear Convergence of the Proximal Gradient Method for Composite Optimization Under the Polyak-Łojasiewicz Inequality and Its Variant

Linear Convergence of Variance-Reduced Stochastic Gradient without Strong Convexity