Abstract:In the linear regression model, the minimum -norm interpolant estimator has received much attention since it was proved to be consistent even though it fits noisy data perfectly under some condition on the covariance matrix of the input vector, known as benign overfitting . Motivated by this phenomenon, we study the generalization property of this estimator from a geometrical viewpoint. Our main results extend and improve the convergence rates as well as the deviation probability from (Tsigler et al. in J Mach Learn Res 24(123):1–76, 2021). Our proof differs from the classical bias/variance analysis and is based on the self-induced regularization property introduced in [5]: can be written as a sum of a ridge estimator and an overfitting component which follows a decomposition of the features space into the space spanned by the top k eigenvectors of and spanned by the last ones. We also prove a matching lower bound for the expected prediction risk thus obtain the sufficient and necessary conditions for benign overfitting of . The two geometrical properties of random Gaussian matrices at the heart of our analysis are the Dvoretsky-Milman theorem and isomorphic and restricted isomorphic properties. In particular, the Dvoretsky dimension appearing naturally in our geometrical viewpoint, coincides with the effective rank from (Bartlett Proc Natl Acad Sci 117(48), 30063–30070, 2020), (Tsigler et al. in J Mach Learn Res 24(123):1–76, 2021) and is the key tool for handling the behavior of the design matrix restricted to the sub-space where overfitting happens. We extend these results to heavy-tailed scenarii proving the universality of this phenomenon beyond exponential moment assumptions. This phenomenon is unknown before and is widely believed to be a significant challenge. This follows from an anistropic version of the probabilistic Dvoretsky-Milman theorem that holds for heavy-tailed vectors which is of independent interest.

The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks

On the Generalization Properties of Minimum-norm Solutions for Over-parameterized Neural Network Models

Do highly over-parameterized neural networks generalize since bad solutions are rare?

Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression

Generalization Error of Generalized Linear Models in High Dimensions

On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models

Generalization Error of Graph Neural Networks in the Mean-field Regime

A priori generalization error for two-layer ReLU neural network through minimum norm solution

The slow deterioration of the generalization error of the random feature model

The Slow Deterioration of the Generalization Error of the Random Feature Model

A priori estimates of the generalization error for two-layer neural networks

Generalization Error Analysis of Neural networks with Gradient Based Regularization

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization

Scaling description of generalization with number of parameters in deep learning

Theory IIIb: Generalization in Deep Networks

A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics

Generalization and Estimation Error Bounds for Model-based Neural Networks

Generalization for Least Squares Regression With Simple Spiked Covariances

On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model

A geometrical viewpoint on the benign overfitting property of the minimum -norm interpolant estimator and its universality