Abstract:Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with infinite width (Luo et al., 2021), we make a step towards drawing a phase diagram for three-layer ReLU NNs with infinite width. First, we derive a normalized gradient flow for three-layer ReLU NNs and obtain two key independent quantities to distinguish different dynamical regimes for common initialization methods. With carefully designed experiments and a large computation cost, for both synthetic datasets and real datasets, we find that the dynamics of each layer also could be divided into a linear regime and a condensed regime, separated by a critical regime. The criteria is the relative change of input weights (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) as the width approaches infinity during the training, which tends to $0$, $+\infty$ and $O(1)$, respectively. In addition, we also demonstrate that different layers can lie in different dynamical regimes in a training process within a deep NN. In the condensed regime, we also observe the condensation of weights in isolated orientations with low complexity. Through experiments under three-layer condition, our phase diagram suggests a complicated dynamical regimes consisting of three possible regimes, together with their mixture, for deep NNs and provides a guidance for studying deep NNs in different initialization regimes, which reveals the possibility of completely different dynamics emerging within a deep NN for its different layers.

Weight decay induced phase transitions in multilayer neural networks

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Statistical physics and practical training of soft-committee machines

Phase Diagram for Two-layer ReLU Neural Networks at Infinite-width Limit.

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks

On the Weight Dynamics of Deep Normalized Networks

On-line learning dynamics of ReLU neural networks using statistical physics techniques

On the weight dynamics of learning networks

Learning Time-Scales in Two-Layers Neural Networks

Catapult Dynamics and Phase Transitions in Quadratic Nets

Dynamical transition in controllable quantum neural networks with large depth

Cascade of phase transitions in the training of Energy-based models

On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages

Multilayer neural networks with extensively many hidden units

Low-dimensional Intrinsic Dimension Reveals a Phase Transition in Gradient-Based Learning of Deep Neural Networks

Consequences of Dale's law on the stability-complexity relationship of random neural networks

Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks

An Analysis of Weight Decay As a Methodology of Reducing Three-Layer Feedforward Artificial Neural Networks for Classification Problems

Phase transitions in the mini-batch size for sparse and dense two-layer neural networks

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

Density of states in neural networks: an in-depth exploration of learning in parameter space