Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Prelimit Coupling and Steady-State Convergence of Constant-stepsize Nonsmooth Contractive SA

Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation

The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize

Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic Approximation

Concentration of Contractive Stochastic Approximation: Additive and Multiplicative Noise

A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequences

Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterization

Decoupled Functional Central Limit Theorems for Two-Time-Scale Stochastic Approximation

Two-Timescale Linear Stochastic Approximation: Constant Stepsizes Go a Long Way

On the Convergence of Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the Convergence Rate

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate.

Nonsmooth Decentralized Stabilization for Interconnected Systems Subject to Strongly Coupled Uncertain Interactions.

Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications

Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent

Temporal approximation of stochastic evolution equations with irregular nonlinearities

Unified ODE Analysis of Smooth Q-Learning Algorithms

Finite-Time Error Bounds of Biased Stochastic Approximation With Application to TD-Learning