Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Online estimation of the asymptotic variance for averaged stochastic gradient algorithms

Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

A Fully Online Approach for Covariance Matrices Estimation of Stochastic Gradient Descent Solutions

Online Covariance Matrix Estimation in Stochastic Gradient Descent

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Variance Reduced Diffusion Adaptation for Online Learning over Networks

Online Statistical Inference for Gradient-free Stochastic Optimization

On the asymptotic rate of convergence of Stochastic Newton algorithms and their Weighted Averaged versions

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Online estimation of the geometric median in Hilbert spaces : non asymptotic confidence balls

Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications

Efficient and fast estimation of the geometric median in Hilbert spaces with an averaged stochastic gradient algorithm

Online stochastic Newton methods for estimating the geometric median and applications

Accelerating stochastic gradient descent using predictive variance reduction

Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm

Limit Theorems for Stochastic Gradient Descent with Infinite Variance

Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reduction

Online covariance estimation for stochastic gradient descent under Markovian sampling

Sampling and Update Frequencies in Proximal Variance-Reduced Stochastic Gradient Methods

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications