Abstract:Deep learning methods operate in regimes that defy the traditional statistical mindset. Neural network architectures often contain more parameters than training samples, and are so rich that they can interpolate the observed labels, even if the latter are replaced by pure noise. Despite their huge complexity, the same architectures achieve small generalization error on real data. This phenomenon has been rationalized in terms of a so-called `double descent' curve. As the model complexity increases, the test error follows the usual U-shaped curve at the beginning, first decreasing and then peaking around the interpolation threshold (when the model achieves vanishing training error). However, it descends again as model complexity exceeds this threshold. The global minimum of the test error is found above the interpolation threshold, often in the extreme overparametrization regime in which the number of parameters is much larger than the number of samples. Far from being a peculiar property of deep neural networks, elements of this behavior have been demonstrated in much simpler settings, including linear regression with random covariates. In this paper we consider the problem of learning an unknown function over the $d$-dimensional sphere $\mathbb S^{d-1}$, from $n$ i.i.d. samples $(\boldsymbol x_i, y_i)\in \mathbb S^{d-1} \times \mathbb R$, $i\le n$. We perform ridge regression on $N$ random features of the form $\sigma(\boldsymbol w_a^{\mathsf T} \boldsymbol x)$, $a\le N$. This can be equivalently described as a two-layers neural network with random first-layer weights. We compute the precise asymptotics of the test error, in the limit $N,n,d\to \infty$ with $N/d$ and $n/d$ fixed. This provides the first analytically tractable model that captures all the features of the double descent phenomenon without assuming ad hoc misspecification structures.

Neural collapse under cross-entropy loss

The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features

Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model

Generalizing and Decoupling Neural Collapse Via Hyperspherical Uniformity Gap

Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data

The Prevalence of Neural Collapse in Neural Multivariate Regression

The Exploration of Neural Collapse under Imbalanced Data

Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse

Geometric Analysis of Unconstrained Feature Models with $d=K$

Towards understanding neural collapse in supervised contrastive learning with the information bottleneck method

Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

On the Robustness of Neural Collapse and the Neural Collapse of Robustness

Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model

Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias

The Usual Suspects? Reassessing Blame for VAE Posterior Collapse

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

Limitations of Neural Collapse for Understanding Generalization in Deep Learning

Prevalence of Neural Collapse during the terminal phase of deep learning training

The generalization error of random features regression: Precise asymptotics and double descent curve