Abstract:This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function, defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; another choice is the output of the celebrated backpropagation algorithm, which is popular amongst practioners, and whose properties have recently been studied by Bolte and Pauwels. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential of the mean function, it has been assumed in the literature that an oracle of the Clarke subdifferential of the mean function is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of a particular differential inclusion: the subgradient flow. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function.

From Stationary Phase to Steepest Descent

New Paradigm for Turbulent Transport Across a Steep Gradient in Toroidal Plasmas.

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

The Nonlinear Steepest Descent Method to Long-Time Asymptotics of the Coupled Nonlinear Schrödinger Equation

Behaviour of the Serre Equations in the Presence of Steep Gradients Revisited

An Analysis of a Multi-Level Projected Steepest Descent Iteration for Nonlinear Inverse Problems in Banach Spaces Subject to Stability Constraints

Minimum Flow Steepest Descent Approach for Nonlinear PDE

Nonlinear steepest descent on a torus: A case study of the Landau-Lifshitz equation

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descent

On Steepest-Descent-Kaczmarz methods for regularizing systems of nonlinear ill-posed equations

SOLUTION OF NON-LINEAR PROBLEMS OF STRUCTURAL MECHANICS BY METHOD OF STEEPEST DESCENT

Phase Space Models for Stochastic Nonlinear Parabolic Waves: Wave Spread and Singularity

THE DISCRETE NONLINEAR SCHRÖDINGER EQUATION: A SURVEY OF RECENT RESULTS

On Degenerate Partial Differential Equations

A semismooth conjugate gradients method – theoretical analysis

Long-time Asymptotic Behavior for the Derivative Schrödinger Equation with Finite Density Type Initial Data

Stationary solutions for the nonlinear Schrödinger equation

Gradient ascent method for fully nonlinear parabolic differential equations with convex nonlinearity

Breather Solutions to a Two-dimensional Nonlinear Schrödinger Equation with Non-local Derivatives

Design of Sturm global attractors 2: Time-reversible Chafee-Infante lattices of 3-nose meanders