A Dynamical Model of Neural Scaling Laws

Blake Bordelon,Alexander Atanasov,Cengiz Pehlevan

2024-06-23

Abstract:On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is the compute-optimal scaling law, which reports the performance as a function of units of compute when choosing model sizes optimally. We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. This reproduces many observations about neural scaling laws. First, our model makes a prediction about why the scaling of performance with training time and with model size have different power law exponents. Consequently, the theory predicts an asymmetric compute-optimal scaling rule where the number of training steps are increased faster than model parameters, consistent with recent empirical observations. Second, it has been observed that early in training, networks converge to their infinite-width dynamics at a rate $1/\textit{width}$ but at late time exhibit a rate $\textit{width}^{-c}$, where $c$ depends on the structure of the architecture and task. We show that our model exhibits this behavior. Lastly, our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.

Machine Learning,Disordered Systems and Neural Networks

What problem does this paper attempt to address?

The paper mainly discusses the dynamic model of neural network scaling law, especially the phenomena related to the influences of training time, model size, and dataset size on performance. In the study, the authors simulate network training and generalization using a randomly featured model trained via gradient descent. The paper presents the following key points:

1. **Relationship between performance and training time and model size**: The test loss increases with different power law exponents as the training time and model size increase, leading to an asymmetric computation-optimal scaling rule where the number of training steps increases faster than the number of model parameters for a given computation budget.

2. **Early and late-stage training dynamics**: The network converges to an infinite width dynamic at a rate of 1/width in the early training stage, but in the later stage, it demonstrates convergence speed related to the architecture and task structure through the width-c relationship.

3. **Training and testing loss gap**: The gap between training and testing loss gradually increases over time due to the reuse of data.

4. **Different scaling exponents with different training times**: Later on, the scaling exponent of the model differs from that of the training time, resulting in an asymmetric computation-optimal strategy.

5. **Larger models train faster**: Keeping the feature learning unchanged, wider networks achieve lower test loss within the same number of iterations.

6. **Limited data and width corrections**: The model accumulates limited data and width corrections during the training process, leading to increasing performance differences over time.

7. **Effect of training time on scaling exponent**: The scaling exponent of task dependency differs in the later time compared to the early time, but the exponent for the early time is universal.

8. **Difference between ensembles and increasing width**: Ensembles (e.g., ensemble learning) are not equivalent to increasing model width, as increasing model size usually provides greater performance improvements.

By studying these phenomena, the paper establishes a mathematically tractable model to simultaneously consider changes in time, model size, and dataset size, aiming to explain the origin and related phenomena of the neural network scaling law.