Abstract:We present a novel algorithm for training deep neural networks in supervised (classification and regression) and unsupervised (reinforcement learning) scenarios. This algorithm combines the standard stochastic gradient descent and the gradient clipping method. The output layer is updated using clipped gradients, the rest of the neural network is updated using standard gradients. Updating the output layer using clipped gradient stabilizes it. We show that the remaining layers are automatically stabilized provided the neural network is only composed of squashing (compact range) activations. We also present a novel squashing activation function - it is obtained by modifying a Gaussian Error Linear Unit (GELU) to have compact range - we call it Truncated GELU (tGELU). Unlike other squashing activations, such as sigmoid, the range of tGELU can be explicitly specified. As a consequence, the problem of vanishing gradients that arise due to a small range, e.g., in the case of a sigmoid activation, is eliminated. We prove that a NN composed of squashing activations (tGELU, sigmoid, etc.), when updated using the algorithm presented herein, is numerically stable and has consistent performance (low variance). The theory is supported by extensive experiments. Within reinforcement learning, as a consequence of our study, we show that target networks in Deep Q-Learning can be omitted, greatly speeding up learning and alleviating memory requirements. Cross-entropy based classification algorithms that suffer from high variance issues are more consistent when trained using our framework. One symptom of numerical instability in training is the high variance of the neural network update values. We show, in theory and through experiments, that our algorithm updates have low variance, and the training loss reduces in a smooth manner.

A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data

PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks

Pseudo inverse versus iterated projection: Novel learning approach and its application on broad learning system

Semi-adaptive Synergetic Two-way Pseudoinverse Learning System

LPGD: A General Framework for Backpropagation through Embedded Optimization Layers

A Pseudoinversion-Free Method for Weight Updating in Broad Learning System

A Pseudoinverse Incremental Algorithm For Fast Training Deep Neural Networks With Application To Spectra Pattern Recognition

A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks

A Progressive Stacking Pseudoinverse Learning Framework via Active Learning in Random Subspaces

A new approach to generalisation error of machine learning algorithms: Estimates and convergence

A Fast Learning Algorithm for Feedfoward Neural Network Based on the Layer-by-Layer and Neuron-by-Neuron Optimizing Procedure

A Theoretical Framework for Inference Learning

Physarum Powered Differentiable Linear Programming Layers and Applications

Inverting Feedforward Neural Networks Using Linear and Nonlinear Programming

Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent

Enhanced Probabilistic Inference Algorithm Using Probabilistic Neural Networks For Learning Control

Optimizing Neural Networks with Gradient Lexicase Selection

Differentiable Forward and Backward Fixed-Point Iteration Layers

Hidden Layer Performance Evaluation for Feedforward Neural Networks Based on Hidden Layer Growing Strategy

Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond

Theory IIIb: Generalization in Deep Networks