Abstract:Recurrent neural networks (RNNs) are popular machine learning tools for modeling and forecasting sequential data and for inferring dynamical systems (DS) from observed time series. Concepts from DS theory (DST) have variously been used to further our understanding of both, how trained RNNs solve complex tasks, and the training process itself. Bifurcations are particularly important phenomena in DS, including RNNs, that refer to topological (qualitative) changes in a system's dynamical behavior as one or more of its parameters are varied. Knowing the bifurcation structure of an RNN will thus allow to deduce many of its computational and dynamical properties, like its sensitivity to parameter variations or its behavior during training. In particular, bifurcations may account for sudden loss jumps observed in RNN training that could severely impede the training process. Here we first mathematically prove for a particular class of ReLU-based RNNs that certain bifurcations are indeed associated with loss gradients tending toward infinity or zero. We then introduce a novel heuristic algorithm for detecting all fixed points and k-cycles in ReLU-based RNNs and their existence and stability regions, hence bifurcation manifolds in parameter space. In contrast to previous numerical algorithms for finding fixed points and common continuation methods, our algorithm provides exact results and returns fixed points and cycles up to high orders with surprisingly good scaling behavior. We exemplify the algorithm on the analysis of the training process of RNNs, and find that the recently introduced technique of generalized teacher forcing completely avoids certain types of bifurcations in training. Thus, besides facilitating the DST analysis of trained RNNs, our algorithm provides a powerful instrument for analyzing the training process itself.

Online Learning Dynamics of Radial Basis Function Neural Networks Near the Singularity

Dynamics of learning near singularities in radial basis function networks.

Eigenvalue Analysis on Singularity in RBF Networks

Numerical Analysis Near Singularities in RBF Networks.

Dynamics of learning near singularities in layered networks.

Theoretical and Numerical Analysis of Learning Dynamics Near Singularity in Multilayer Perceptrons

A Novel Nonlinear Dynamic System Modeling Approach Using Radial Basis Function Neural Networks

Gradient Learning Dynamics of Radial Basis Function Networks

Dynamics of Learning In Hierarchical Models – Singularity and Milnor Attractor

Influence Area of Overlap Singularity in Multilayer Perceptrons

Research on RBF neural network method of singularity detection in chaotic time series

A Novel Radial Basis Function Neural Network For Approximation

Rethinking Normalization and Elimination Singularity in Neural Networks

Does SGD really happen in tiny subspaces?

Singularities in the Parameter Identification of Nonlinear Dynamic Systems

Stochastic Bifurcations and Transient Dynamics of Probability Responses with Radial Basis Function Neural Networks

Consideration on the learning efficiency of multiple-layered neural networks with linear units

Bifurcations and loss jumps in RNN training

Gradient dynamics for low-rank fine-tuning beyond kernels

Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos

Learning Time-Scales in Two-Layers Neural Networks