Abstract:Stochastic gradient descent (SGD) optimization methods such as the plain vanilla SGD method and the popular Adam optimizer are nowadays the method of choice in the training of artificial neural networks (ANNs). Despite the remarkable success of SGD methods in the ANN training in numerical simulations, it remains in essentially all practical relevant scenarios an open problem to rigorously explain why SGD methods seem to succeed to train ANNs. In particular, in most practically relevant supervised learning problems, it seems that SGD methods do with high probability not converge to global minimizers in the optimization landscape of the ANN training problem. Nevertheless, it remains an open problem of research to disprove the convergence of SGD methods to global minimizers. In this work we solve this research problem in the situation of shallow ANNs with the rectified linear unit (ReLU) and related activations with the standard mean square error loss by disproving in the training of such ANNs that SGD methods (such as the plain vanilla SGD, the momentum SGD, the AdaGrad, the RMSprop, and the Adam optimizers) can find a global minimizer with high probability. Even stronger, we reveal in the training of such ANNs that SGD methods do with high probability fail to converge to global minimizers in the optimization landscape. The findings of this work do, however, not disprove that SGD methods succeed to train ANNs since they do not exclude the possibility that SGD methods find good local minimizers whose risk values are close to the risk values of the global minimizers. In this context, another key contribution of this work is to establish the existence of a hierarchical structure of local minimizers with distinct risk values in the optimization landscape of ANN training problems with ReLU and related activations.

Globally Convergent Neural Networks

Adaptive Deep Spiking Neural Network with Global-Local Learning Via Balanced Excitatory and Inhibitory Mechanism

Neural Networks: Computational Models and Applications - ReadingSample

Training Artificial Neural Networks Using a Global Optimization Method That Utilizes Neural Networks

A Neural Network Transformation based Global Optimization Algorithm

Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks

A Novel Global Training Algorithm and Its Convergence Theorem for Fuzzy Neural Networks

A Recipe for Global Convergence Guarantee in Deep Neural Networks

Optimal Training of Feedforward Neural Networks Using Teaching-Learning-Based Optimization with Modified Learning Phases

A Globally Convergent Algorithm for Neural Network Parameter Optimization Based on Difference-of-Convex Functions

GLSNN: A Multi-Layer Spiking Neural Network Based on Global Feedback Alignment and Local STDP Plasticity

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Ada-GNN: Adapting to Local Patterns for Improving Graph Neural Networks

Design and Analysis of a Novel Distributed Gradient Neural Network for Solving Consensus Problems in a Predefined Time

Predict globally, correct locally: Parallel-in-time optimization of neural networks

Optimal ANN-SNN Conversion with Group Neurons

The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses

Genetic-GNN: Evolutionary architecture search for Graph Neural Networks

On Convergence of Training Loss Without Reaching Stationary Points

On the Convergence and Improvement of Stochastic Normalized Gradient Descent

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks