Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

A new convergence rate of the steepest descent regarding the Euclidean norm

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Riemannian Steepest Descent

On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions

The Average Rate of Convergence of the Exact Line Search Gradient Descent Method

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

Convergence rate analysis of the gradient descent–ascent method for convex–concave saddle-point problems

On a Faster $R$-Linear Convergence Rate of the Barzilai-Borwein Method

Linear Convergence of Adaptive Stochastic Gradient Descent

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Linearly convergent adjoint free solution of least squares problems by random descent

Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

R-Linear Convergence of Limited Memory Steepest Descent

The Randomized Block Coordinate Descent Method in the Hölder Smooth Setting

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints

Faster Acceleration for Steepest Descent

Linear convergence of first order methods for non-strongly convex optimization