Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Entropy-based convergence rates of greedy algorithms

Entropy-based convergence rates of greedy algorithms

A new analysis of empirical interpolation methods and Chebyshev greedy algorithms

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

On the rate of convergence of greedy algorithms

Some New Convergence Analysis and Applications of POD-Greedy Algorithms

Convergence and rate of convergence of some greedy algorithms in convex optimization

Rate of convergence of Thresholding Greedy Algorithms

On the convergence analysis of the greedy randomized Kaczmarz method

A unified way of analyzing some greedy algorithms

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Greedy Algorithms for Reduced Bases in Banach Spaces

Biorthogonal Greedy Algorithms in convex optimization

On the entropy numbers of the mixed smoothness function classes

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Non-linear approximation by $1$-greedy bases

Embeddings and Lebesgue-Type Inequalities for the Greedy Algorithm in Banach Spaces

Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularization