Abstract:Numerous theories of learning propose to prevent the gradient from exponential growth with depth or time, to stabilize and improve training. Typically, these analyses are conducted on feed-forward fully-connected neural networks or simple single-layer recurrent neural networks, given their mathematical tractability. In contrast, this study demonstrates that pre-training the network to local stability can be effective whenever the architectures are too complex for an analytical initialization. Furthermore, we extend known stability theories to encompass a broader family of deep recurrent networks, requiring minimal assumptions on data and parameter distribution, a theory we call the Local Stability Condition (LSC). Our investigation reveals that the classical Glorot, He, and Orthogonal initialization schemes satisfy the LSC when applied to feed-forward fully-connected neural networks. However, analysing deep recurrent networks, we identify a new additive source of exponential explosion that emerges from counting gradient paths in a rectangular grid in depth and time. We propose a new approach to mitigate this issue, that consists on giving a weight of a half to the time and depth contributions to the gradient, instead of the classical weight of one. Our empirical results confirm that pre-training both feed-forward and recurrent networks, for differentiable, neuromorphic and state-space models to fulfill the LSC, often results in improved final performance. This study contributes to the field by providing a means to stabilize networks of any complexity. Our approach can be implemented as an additional step before pre-training on large augmented datasets, and as an alternative to finding stable initializations analytically.

Orthogonal Gated Recurrent Unit with Neumann-Cayley Transformation

Residual Recurrent Neural Networks for Learning Sequential Representations.

Orthogonal Graph Neural Networks

Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Stabilizing RNN Gradients through Pre-training

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

Simple Orthogonal Graph Representation Learning (student Abstract)

Gradient Projection Memory for Continual Learning

Orthogonal Gradient Descent for Continual Learning

Restricted Orthogonal Gradient Projection for Continual Learning

Classical and Quantum Algorithms for Orthogonal Neural Networks

Introducing Common Null Space of Gradients for Gradient Projection Methods in Continual Learning

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

Neural Arithmetic Logic Units with Two Transition Matrix and Independent Gates

NDOT: Neuronal Dynamics-based Online Training for Spiking Neural Networks

An empirical exploration of recurrent network architectures

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Gates Are Not What You Need in RNNs

A Cerebellum Model Enhanced Gradient Neural Network for Solving Time-Variant Linear Equations Applied to Robot Control

A Novel Lightweight Grouped Gated Recurrent Unit for Automatic Modulation Classification