Abstract:Deep learning methods operate in regimes that defy the traditional statistical mindset. Neural network architectures often contain more parameters than training samples, and are so rich that they can interpolate the observed labels, even if the latter are replaced by pure noise. Despite their huge complexity, the same architectures achieve small generalization error on real data. This phenomenon has been rationalized in terms of a so-called `double descent' curve. As the model complexity increases, the test error follows the usual U-shaped curve at the beginning, first decreasing and then peaking around the interpolation threshold (when the model achieves vanishing training error). However, it descends again as model complexity exceeds this threshold. The global minimum of the test error is found above the interpolation threshold, often in the extreme overparametrization regime in which the number of parameters is much larger than the number of samples. Far from being a peculiar property of deep neural networks, elements of this behavior have been demonstrated in much simpler settings, including linear regression with random covariates. In this paper we consider the problem of learning an unknown function over the $d$-dimensional sphere $\mathbb S^{d-1}$, from $n$ i.i.d. samples $(\boldsymbol x_i, y_i)\in \mathbb S^{d-1} \times \mathbb R$, $i\le n$. We perform ridge regression on $N$ random features of the form $\sigma(\boldsymbol w_a^{\mathsf T} \boldsymbol x)$, $a\le N$. This can be equivalently described as a two-layers neural network with random first-layer weights. We compute the precise asymptotics of the test error, in the limit $N,n,d\to \infty$ with $N/d$ and $n/d$ fixed. This provides the first analytically tractable model that captures all the features of the double descent phenomenon without assuming ad hoc misspecification structures.

Mean Dimension of Ridge Functions

Approximation of generalized ridge functions in high dimensions

Dimension free ridge regression

Dimension Reduction via Gaussian Ridge Functions

On Ridge Estimation in High-dimensional Rotationally Sparse Linear Regression

Asymptotic Confidence Regions for Density Ridges

Effect of dimensionality on convergence rates of kernel ridge regression estimator

The generalization error of random features regression: Precise asymptotics and double descent curve

High-dimensional analysis of double descent for linear regression with random projections

Highly Adaptive Ridge

On the approximation by weighted ridge functions

Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view

Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions

On the Universality of the Double Descent Peak in Ridgeless Regression

Notes on the dimension dependence in high-dimensional central limit theorems for hyperrectangles

Deep sections of the hypercube

Scaling and renormalization in high-dimensional regression

Independence of synthetic Curvature Dimension conditions on transport distance exponent

A Bound on the Maximal Marginal Degrees of Freedom

A sparsified Christoffel function for high-dimensional inference

On best approximation by multivariate ridge functions with applications to generalized translation networks