Abstract:In this paper, we study the convergence properties of the Stochastic Gradient Descent (SGD) method for finding a stationary point of a given objective function $J(\cdot)$. The objective function is not required to be convex. Rather, our results apply to a class of ``invex'' functions, which have the property that every stationary point is also a global minimizer. First, it is assumed that $J(\cdot)$ satisfies a property that is slightly weaker than the Kurdyka-Lojasiewicz (KL) condition, denoted here as (KL'). It is shown that the iterations $J(\boldsymbol{\theta}_t)$ converge almost surely to the global minimum of $J(\cdot)$. Next, the hypothesis on $J(\cdot)$ is strengthened from (KL') to the Polyak-Lojasiewicz (PL) condition. With this stronger hypothesis, we derive estimates on the rate of convergence of $J(\boldsymbol{\theta}_t)$ to its limit. Using these results, we show that for functions satisfying the PL property, the convergence rate of both the objective function and the norm of the gradient with SGD is the same as the best-possible rate for convex functions. While some results along these lines have been published in the past, our contributions contain two distinct improvements. First, the assumptions on the stochastic gradient are more general than elsewhere, and second, our convergence is almost sure, and not in expectation. We also study SGD when only function evaluations are permitted. In this setting, we determine the ``optimal'' increments or the size of the perturbations. Using the same set of ideas, we establish the global convergence of the Stochastic Approximation (SA) algorithm under more general assumptions on the measurement error, compared to the existing literature. We also derive bounds on the rate of convergence of the SA algorithm under appropriate assumptions.

Stochastic Gradient Descent for Linear Inverse Problems in Hilbert Spaces.

Stochastic gradient descent for linear inverse problems in variable exponent Lebesgue spaces

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

Stochastic variance reduced gradient method for linear ill-posed inverse problems

Statistical Learning and Inverse Problems: A Stochastic Gradient Approach

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

Stochastic Data-Driven Bouligand Landweber Method for Solving Non-smooth Inverse Problems

Learning Operators with Stochastic Gradient Descent in General Hilbert Spaces

Stochastic Gradient Descent Revisited

The Practicality of Stochastic Optimization in Imaging Inverse Problems

On the Discrepancy Principle for Stochastic Gradient Descent

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent

Diagonalisation SGD: Fast & Convergent SGD for Non-Differentiable Models via Reparameterisation and Smoothing

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions

Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization

Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation