Abstract:We present a novel algorithm for training deep neural networks in supervised (classification and regression) and unsupervised (reinforcement learning) scenarios. This algorithm combines the standard stochastic gradient descent and the gradient clipping method. The output layer is updated using clipped gradients, the rest of the neural network is updated using standard gradients. Updating the output layer using clipped gradient stabilizes it. We show that the remaining layers are automatically stabilized provided the neural network is only composed of squashing (compact range) activations. We also present a novel squashing activation function - it is obtained by modifying a Gaussian Error Linear Unit (GELU) to have compact range - we call it Truncated GELU (tGELU). Unlike other squashing activations, such as sigmoid, the range of tGELU can be explicitly specified. As a consequence, the problem of vanishing gradients that arise due to a small range, e.g., in the case of a sigmoid activation, is eliminated. We prove that a NN composed of squashing activations (tGELU, sigmoid, etc.), when updated using the algorithm presented herein, is numerically stable and has consistent performance (low variance). The theory is supported by extensive experiments. Within reinforcement learning, as a consequence of our study, we show that target networks in Deep Q-Learning can be omitted, greatly speeding up learning and alleviating memory requirements. Cross-entropy based classification algorithms that suffer from high variance issues are more consistent when trained using our framework. One symptom of numerical instability in training is the high variance of the neural network update values. We show, in theory and through experiments, that our algorithm updates have low variance, and the training loss reduces in a smooth manner.

Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods

Trinity: Neural Network Adaptive Distributed Parallel Training Method Based on Reinforcement Learning.

A separable training algorithm based on nonmonotone trust-region method for Neural ODE

A Two-Stage Subspace Trust Region Approach for Deep Neural Network Training

A novel consensus PSO-assisted trajectory unified and trust-tech methodology for DNN training and its applications

Predict globally, correct locally: Parallel-in-time optimization of neural networks

Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximations

Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint

Dynamic Universal Approximation Theory: Foundations for Parallelism in Neural Networks

Training Artificial Neural Networks Using a Global Optimization Method That Utilizes Neural Networks

Towards Scalable and Stable Parallelization of Nonlinear RNNs

Training Neural Networks Using Reproducing Kernel Space Interpolation and Model Reduction

AxTrain

Training-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion

Training Neural Networks from Scratch with Parallel Low-Rank Adapters

Partitioned integrators for thermodynamic parameterization of neural networks

A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks

Preconditioners for the Stochastic Training of Implicit Neural Representations

Parallel Algorithms Align with Neural Execution

Parallelization Techniques for Verifying Neural Networks

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform