Scaling description of generalization with number of parameters in deep learning
Mario Geiger,Arthur Jacot,Stefano Spigler,Franck Gabriel,Levent Sagun,Stéphane d’Ascoli,Giulio Biroli,Clément Hongler,Matthieu Wyart
DOI: https://doi.org/10.1088/1742-5468/ab633c
2020-02-05
Abstract:Supervised deep learning involves the training of neural networks with a large number <i>N</i> of parameters. For large enough <i>N</i>, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as <i>N</i> grows past a certain threshold <i>N</i> <sup>*</sup>. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with <i>N</i>. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations of the neural net output function <i>f</i> <sub> <i>N</i> </sub> around its expectation . These affect the generalization error for classification: under natural assumptions, it decays to a plateau value in a power-law fashion ~<i>N</i> <sup>−1/2</sup>. This description breaks down at a so-called jamming transition <i>N</i> = <i>N</i> <sup>*</sup>. At this threshold, we argue that diverges. This result leads to a plausible explanation for the cusp in test error known to occur at <i>N</i> <sup>*</sup>. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond <i>N</i> <sup>*</sup>, and averaging their outputs.
physics, mathematical,mechanics