Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Consistency of sample-based stationary points for infinite-dimensional stochastic optimization

Sample Size Estimates for Risk-Neutral Semilinear PDE-Constrained Optimization

On quantitative stability in infinite-dimensional optimization under uncertainty

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Stability and Sample-based Approximations of Composite Stochastic Optimization Problems

Solving Nonsmooth Nonconvex Compound Stochastic Programs with Applications to Risk Measure Minimization

Sample Average Approximation Method for a Class of Stochastic Variational Inequality Problems.

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

A stochastic regularized second-order iterative scheme for optimal control and inverse problems in stochastic partial differential equations

A relaxation-based probabilistic approach for PDE-constrained optimization under uncertainty with pointwise state constraints

Sample Average Approximation for Stochastic Programming with Equality Constraints

Robust Stochastic Approximation Approach to Stochastic Programming

Statistical inference and hypotheses testing of risk averse stochastic programs

Criticality measure-based error estimates for infinite dimensional optimization

Stochastic approximation in infinite dimensions

Nonasymptotic Upper Estimates for Errors of the Sample Average Approximation Method to Solve Risk-Averse Stochastic Programs

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Goldstein Stationarity in Lipschitz Constrained Optimization

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Randomized quasi-Monte Carlo methods for risk-averse stochastic optimization