Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Convergence rates in the Prokhorov metric for assessing uncertainty in ill-posed problems

Convergence rates for critical point regularization

Convergence Rate Analysis of Tikhonov Regularization for Nonlinear Ill-Posed Problems with Noisy Operators

On convergence and convergence rates for Ivanov and Morozov regularization and application to some parameter identification problems in elliptic PDEs

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Convergence rate of inexact proximal point algorithms for operator with holder metric subregularity

Generalized Convergence Rates Results for Linear Inverse Problems in Hilbert Spaces

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

On the convergence of an inertial proximal algorithm with a Tikhonov regularization term

New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

Strong convergence and fast rates for systems with Tikhonov regularization

Convergence Rates for the Maximum A Posteriori Estimator in PDE-Regression Models with Random Design

Preasymptotic convergence of randomized Kaczmarz method

Stochastic Convergence of Regularized Solutions and Their Finite Element Approximations to Inverse Source Problems

Beyond convergence rates: Exact recovery with Tikhonov regularization with sparsity constraints

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Convergence Analysis of a Proximal Stochastic Denoising Regularization Algorithm

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

Convergence analysis of equilibrium methods for inverse problems

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications