Abstract:This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function, defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; another choice is the output of the celebrated backpropagation algorithm, which is popular amongst practioners, and whose properties have recently been studied by Bolte and Pauwels. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential of the mean function, it has been assumed in the literature that an oracle of the Clarke subdifferential of the mean function is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of a particular differential inclusion: the subgradient flow. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function.

Convergence of a double step scheme for a class of second order Clarke subdifferential inclusions

Convergence of double step scheme for a class of parabolic Clarke subdifferential inclusions

Convergence of a Time Discretization for Nonlinear Second Order Inclusion

Evolution inclusions with Clarke subdifferential type in Hilbert space

Convergence Analysis of a Method for Variational Inclusions

Convergence of the family of the deformed Euler–Halley iterations under the Hölder condition of the second derivative

Regularity Analysis and High-Order Time Stepping Scheme for Quasilinear Subdiffusion

Constant Step Stochastic Approximations Involving Differential Inclusions: Stability, Long-Run Convergence and Applications

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

A new kind of double phase elliptic inclusions with logarithmic perturbation terms II: Applications

Mathematical Analysis and a Second-Order Compact Scheme for Nonlinear Caputo–Hadamard Fractional Sub-diffusion Equations

Convergence and Superconvergence Analysis of a Nonconforming Finite Element Variable-Time-Step BDF2 Implicit Scheme for Linear Reaction-Diffusion Equations

Decoupling multistep schemes for elliptic-parabolic problems

A Class of History-Dependent Systems of Evolution Inclusions with Applications

Second-Order Time Stepping Scheme Combined with a Mixed Element Method for a 2D Nonlinear Fourth-Order Fractional Integro-Differential Equations

Convergence analysis of the maximum principle preserving BDF2 scheme with variable time-steps for the space fractional Allen-Cahn equation

Rothe Method for Parabolic Variational–hemivariational Inequalities

Stability analysis of a second-order difference scheme for the time-fractional mixed sub-diffusion and diffusion-wave equation

A Comparative Study of Efficient Numerical Schemes for Time-Fractional Subdiffusion Equation Involving Singularity

A finite difference scheme for two-dimensional singularly perturbed convection-diffusion problem with discontinuous source term

Second-order differential inclusions with two small parameters