Abstract:This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function, defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; another choice is the output of the celebrated backpropagation algorithm, which is popular amongst practioners, and whose properties have recently been studied by Bolte and Pauwels. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential of the mean function, it has been assumed in the literature that an oracle of the Clarke subdifferential of the mean function is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of a particular differential inclusion: the subgradient flow. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function.

Global convergence of the gradient method for functions definable in o-minimal structures

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

The gradient's limit of a definable family of functions is a conservative set-valued field

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Local Conditions for Global Convergence of Gradient Flows and Proximal Point Sequences in Metric Spaces

Global stability of first-order methods for coercive tame functions

Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradients

Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices

Concavifiability and convergence: necessary and sufficient conditions for gradient descent analysis

On a Family of Relaxed Gradient Descent Methods for Quadratic Minimization

Goldstein Stationarity in Lipschitz Constrained Optimization

Super Gradient Descent: Global Optimization requires Global Gradient

Convergence of Markov Chains for Constant Step-size Stochastic Gradient Descent with Separable Functions

Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networks

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions

Convergence of Nonmonotone Proximal Gradient Methods under the Kurdyka-Lojasiewicz Property without a Global Lipschitz Assumption

Provably Faster Gradient Descent via Long Steps

On the Convergence of Gradient Descent for Large Learning Rates

A Strengthened Conjecture on the Minimax Optimal Constant Stepsize for Gradient Descent

Fast convergence to non-isolated minima: four equivalent conditions for $\mathrm{C}^2$ functions