Abstract:Gradient descent (GD) type optimization methods are the standard instrument to train artificial neural networks (ANNs) with rectified linear unit (ReLU) activation. Despite the great success of GD type optimization methods in numerical simulations for the training of ANNs with ReLU activation, it remains - even in the simplest situation of the plain vanilla GD optimization method with random initializations and ANNs with one hidden layer - an open problem to prove (or disprove) the conjecture that the risk of the GD optimization method converges in the training of such ANNs to zero as the width of the ANNs, the number of independent random initializations, and the number of GD steps increase to infinity. In this article we prove this conjecture in the situation where the probability distribution of the input data is equivalent to the continuous uniform distribution on a compact interval, where the probability distributions for the random initializations of the ANN parameters are standard normal distributions, and where the target function under consideration is continuous and piecewise affine linear. Roughly speaking, the key ingredients in our mathematical convergence analysis are (i) to prove that suitable sets of global minima of the risk functions are \emph{twice continuously differentiable submanifolds of the ANN parameter spaces}, (ii) to prove that the Hessians of the risk functions on these sets of global minima satisfy an appropriate \emph{maximal rank condition}, and, thereafter, (iii) to apply the machinery in [Fehrman, B., Gess, B., Jentzen, A., Convergence rates for the stochastic gradient descent method for non-convex objective functions. J. Mach. Learn. Res. 21(136): 1--48, 2020] to establish convergence of the GD optimization method with random initializations.

Convergence of Hyperbolic Neural Networks under Riemannian Stochastic Gradient Descent

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

Convergence of Riemannian Stochastic Gradient Descent on Hadamard Manifold

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Convergence of stochastic gradient descent under a local Lojasiewicz condition for deep neural networks

Non-asymptotic convergence analysis of the stochastic gradient Hamiltonian Monte Carlo algorithm with discontinuous stochastic gradient with applications to training of ReLU neural networks

Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics

Novel Convergence Results of Adaptive Stochastic Gradient Descents

On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks

Linear Convergence of Adaptive Stochastic Gradient Descent

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Provable convergence of Nesterov’s accelerated gradient method for over-parameterized neural networks

On Convergence of Training Loss Without Reaching Stationary Points

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Almost sure convergence rates of stochastic gradient methods under gradient domination

A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions