Abstract:This paper studies the asymptotic behavior of the constant step Stochastic Gradient Descent for the minimization of an unknown function, defined as the expectation of a non convex, non smooth, locally Lipschitz random function. As the gradient may not exist, it is replaced by a certain operator: a reasonable choice is to use an element of the Clarke subdifferential of the random function; another choice is the output of the celebrated backpropagation algorithm, which is popular amongst practioners, and whose properties have recently been studied by Bolte and Pauwels. Since the expectation of the chosen operator is not in general an element of the Clarke subdifferential of the mean function, it has been assumed in the literature that an oracle of the Clarke subdifferential of the mean function is available. As a first result, it is shown in this paper that such an oracle is not needed for almost all initialization points of the algorithm. Next, in the small step size regime, it is shown that the interpolated trajectory of the algorithm converges in probability (in the compact convergence sense) towards the set of solutions of a particular differential inclusion: the subgradient flow. Finally, viewing the iterates as a Markov chain whose transition kernel is indexed by the step size, it is shown that the invariant distribution of the kernel converge weakly to the set of invariant distribution of this differential inclusion as the step size tends to zero. These results show that when the step size is small, with large probability, the iterates eventually lie in a neighborhood of the critical points of the mean function.

A Necessary and Sufficient Condition for Convergence of Steepest Descent Approximation to Accretive Operator Equations

A Further Necessary and Sufficient Condition for Strong Convergence of Nonlinear Contraction Semigroups and of Iterative Methods for Accretive Operators in Banach Spaces

Weak Convergence of an Iterative Algorithm for Accretive Operators

Sufficient and necessary condition for the convergence of stochastic approximation algorithms

Convergence theorems of the steepest descent method for accretive mappings

On computational feasibility of successive approximation to a solution of nonlinear operator equations

Strong Convergence Theorems for Relatively Nonexpansive Mappings and Variational Inclusions in Banach Spaces

A Strong Convergence Theorem for Solutions of Equilibrium Problems and Asymptotically Quasi-Φ-nonexpansive Mappings in the Intermediate Sense

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Convergence of Newton'S Method for Systems of Equations with Constant Rank Derivatives

A Unified Convergence Theory for Newton-Type Methods for Zeros of Nonlinear Operators in Banach Spaces

Convergence rate of inexact proximal point algorithms for operator with holder metric subregularity

Concavifiability and convergence: necessary and sufficient conditions for gradient descent analysis

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the Convergence Rate

Strong Convergence of an Iterative Algorithm Involving Nonlinear Mappings of Nonexpansive and Accretive Type

Iterative Methods for Zero Points of Accretive Operators in Banach Spaces

Asymptotics for slowly converging evolution equations

A Unified Analysis for the Subgradient Methods Minimizing Composite Nonconvex, Nonsmooth and Non-Lipschitz Functions.

The Existence and Uniqueness of Solution and the Convergence of a Multi-Step Iterative Algorithm for a System of Variational Inclusions with (A, Η, M)-Accretive Operators