Abstract:The Ridgeless minimum $\ell_2$-norm interpolator in overparametrized linear regression has attracted considerable attention in recent years. While it seems to defy the conventional wisdom that overfitting leads to poor prediction, recent research reveals that its norm minimizing property induces an `implicit regularization' that helps prediction in spite of interpolation. This renders the Ridgeless interpolator a theoretically tractable proxy that offers useful insights into the mechanisms of modern machine learning methods. This paper takes a different perspective that aims at understanding the precise stochastic behavior of the Ridgeless interpolator as a statistical estimator. Specifically, we characterize the distribution of the Ridgeless interpolator in high dimensions, in terms of a Ridge estimator in an associated Gaussian sequence model with positive regularization, which plays the role of the prescribed implicit regularization in the context of prediction risk. Our distributional characterizations hold for general random designs and extend uniformly to positively regularized Ridge estimators. As a demonstration of the analytic power of these characterizations, we derive approximate formulae for a general class of weighted $\ell_q$ risks for Ridge(less) estimators that were previously available only for $\ell_2$. Our theory also provides certain further conceptual reconciliation with the conventional wisdom: given any data covariance, a certain amount of regularization in Ridge regression remains beneficial for `most' signals across various statistical tasks including prediction, estimation and inference, as long as the noise level is non-trivial. Surprisingly, optimal tuning can be achieved simultaneously for all the designated statistical tasks by a single generalized or $k$-fold cross-validation scheme, despite being designed specifically for tuning prediction risk.

Ridgeless Regression with Random Features

Ridgelet Kernel Regression

Effective Distributed Learning with Random Features: Improved Bounds and Algorithms

A novel ridgelet kernel regression method

Dimension free ridge regression

A Ridgelet Kernel Regression Model Using Genetic Algorithm.

Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration

Towards Theoretical Understanding of Learning Large-scale Dependent Data Via Random Features

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Kernel Ridge Regression for General Noise Model with Its Application.

Decentralized Kernel Ridge Regression Based on Data-Dependent Random Feature

The generalization error of random features regression: Precise asymptotics and double descent curve

Estimation of Error Variance Via Ridge Regression

The distribution of Ridgeless least squares interpolators

Surprises in high-dimensional ridgeless least squares interpolation

A Ridgelet Kernel Approach for Regression Using Particle Swarm Optimization Algorithm

On the Universality of the Double Descent Peak in Ridgeless Regression

Distributed Randomized Sketching Kernel Learning

Low-rank kernel regression with preserved locality for multi-class analysis

A Duality Analysis of Kernel Ridge Regression in the Noiseless Regime

Optimal Kernel Quantile Learning with Random Features