Abstract:This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function, defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; another choice is the output of the celebrated backpropagation algorithm, which is popular amongst practioners, and whose properties have recently been studied by Bolte and Pauwels. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential of the mean function, it has been assumed in the literature that an oracle of the Clarke subdifferential of the mean function is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of a particular differential inclusion: the subgradient flow. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function.

Steepest Geometric Descent for Regularized Quasiconvex Functions

Steepest geometric descent for regularized quasiconvex functions

On Steepest-Descent-Kaczmarz methods for regularizing systems of nonlinear ill-posed equations

Gauss–Newton Method for Convex Composite Optimizations on Riemannian Manifolds

A new convergence rate of the steepest descent regarding the Euclidean norm

Nonsmooth Steepest Descent Method by Proximal Subdifferentials in Hilbert Spaces

Majorizing Functions and Convergence of the Gauss-Newton Method for Convex Composite Optimization

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Regularized Asymptotic Descents for a Class of Nonconvex Optimization Problems.

Regularity for quasilinear equations on degenerate singular sets

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

On the curve diffusion flow of closed plane curves

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

Characterizations, Dynamical Systems and Gradient Methods for Strongly Quasiconvex Functions

Regularized asymptotic descents: finding the global minima for a class of multiple minima problems

Faster Acceleration for Steepest Descent

Local Curvature Descent: Squeezing More Curvature out of Standard and Polyak Gradient Descent

A PDE approach to the existence and regularity of surfaces of minimum mean curvature variation

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

Accelerated Mirror Descent for Non-Euclidean Star-convex Functions