Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Convergence Rates of Online Critic Value Function Approximation in Native Spaces

Rates of Convergence in Certain Native Spaces of Approximations used in Reinforcement Learning

Convergence analysis of online algorithms for vector-valued kernel regression

Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation

Weak Convergence Analysis of Online Neural Actor-Critic Algorithms

Two-Timescale Critic-Actor for Average Reward MDPs with Function Approximation

Convergence Rates for Localized Actor-Critic in Networked Markov Potential Games

Finite-Sample Analysis of Off-Policy Natural Actor–Critic With Linear Function Approximation

Hypothesis Sketching for Online Kernel Selection in Continuous Kernel Space

Fast and Strong Convergence of Online Learning Algorithms.

Gaussian-kernel-based Adaptive Critic Design Using Two-Phase Value Iteration.

Criticality measure-based error estimates for infinite dimensional optimization

Approximation beats concentration? An approximation view on inference with smooth radial kernels

Error Bounds for a Kernel-Based Constrained Optimal Smoothing Approximation

Actor-Critic or Critic-Actor? A Tale of Two Time Scales

Nearly optimal stochastic approximation for online principal subspace estimation

On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost

Convergence of Unregularized Online Learning Algorithms

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Two-Phase Iteration for Value Function Approximation and Hyperparameter Optimization in Gaussian-Kernel-Based Adaptive Critic Design

Almost-sure quasi-optimal approximation in reproducing kernel Hilbert spaces