Abstract:This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function, defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; another choice is the output of the celebrated backpropagation algorithm, which is popular amongst practioners, and whose properties have recently been studied by Bolte and Pauwels. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential of the mean function, it has been assumed in the literature that an oracle of the Clarke subdifferential of the mean function is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of a particular differential inclusion: the subgradient flow. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function.

The best approximation of a given function in L 2 -norm by Lipschitz functions with gradient constraint

Avoidance of the Lavrentiev gap for one-dimensional non-autonomous functionals with constraints

Goldstein Stationarity in Lipschitz Constrained Optimization

Globally Lipschitz minimizers for variational problems with linear growth

Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

Approximations of Lipschitz maps with maximal derivatives on Banach spaces

Approximation of Lipschitz functions preserving boundary values

An asymptotic viscosity selection result for the regularized Newton dynamic

Smooth approximations preserving asymptotic Lipschitz bounds

Accelerated method of finding for the minimum of arbitrary Lipschitz convex function

On De Giorgi's Conjecture of Nonlocal approximations for free-discontinuity problems: The symmetric gradient case

Approximation of the least Rayleigh quotient for degree $p$ homogeneous functionals

Constrained optimization in classes of analytic functions with prescribed pointwise values

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Convergence and non-convergence in a nonlocal gradient flow

Local Conditions for Global Convergence of Gradient Flows and Proximal Point Sequences in Metric Spaces

Estimating a Function and Its Derivatives Under a Smoothness Condition

Variational problems with gradient constraints: $\textit{A priori}$ and $\textit{a posteriori}$ error identities

Novel projected gradient methods for solving pseudomontone variational inequality problems with applications to optimal control problems

Approximation by mappings with singular Hessian minors