Abstract:The ability of neural networks to provide `best in class' approximation across a wide range of applications is well-documented. Nevertheless, the powerful expressivity of neural networks comes to naught if one is unable to effectively train (choose) the parameters defining the network. In general, neural networks are trained by gradient descent type optimization methods, or a stochastic variant thereof. In practice, such methods result in the loss function decreases rapidly at the beginning of training but then, after a relatively small number of steps, significantly slow down. The loss may even appear to stagnate over the period of a large number of epochs, only to then suddenly start to decrease fast again for no apparent reason. This so-called plateau phenomenon manifests itself in many learning tasks. The present work aims to identify and quantify the root causes of plateau phenomenon. No assumptions are made on the number of neurons relative to the number of training data, and our results hold for both the lazy and adaptive regimes. The main findings are: plateaux correspond to periods during which activation patterns remain constant, where activation pattern refers to the number of data points that activate a given neuron; quantification of convergence of the gradient flow dynamics; and, characterization of stationary points in terms solutions of local least squares regression lines over subsets of the training data. Based on these conclusions, we propose a new iterative training method, the Active Neuron Least Squares (ANLS), characterised by the explicit adjustment of the activation pattern at each step, which is designed to enable a quick exit from a plateau. Illustrative numerical examples are included throughout.

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Phase Diagram for Two-layer ReLU Neural Networks at Infinite-width Limit.

Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training

Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance

Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization

Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages

Topological obstruction to the training of shallow ReLU neural networks

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

A theoretical framework for deep locally connected ReLU network

Normalized gradient flow optimization in the training of ReLU artificial neural networks

Training a Two Layer ReLU Network Analytically

Benign Overfitting for Two-layer ReLU Convolutional Neural Networks

Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics

Catapult Dynamics and Phase Transitions in Quadratic Nets

Understanding Optimization in Deep Learning with Central Flows

On the Banach Spaces Associated with Multi-Layer ReLU Networks: Function Representation, Approximation Theory and Gradient Descent Dynamics

The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks

Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

Benign Overfitting for Regression with Trained Two-Layer ReLU Networks