Abstract:The Ridgeless minimum $\ell_2$-norm interpolator in overparametrized linear regression has attracted considerable attention in recent years. While it seems to defy the conventional wisdom that overfitting leads to poor prediction, recent research reveals that its norm minimizing property induces an `implicit regularization' that helps prediction in spite of interpolation. This renders the Ridgeless interpolator a theoretically tractable proxy that offers useful insights into the mechanisms of modern machine learning methods. This paper takes a different perspective that aims at understanding the precise stochastic behavior of the Ridgeless interpolator as a statistical estimator. Specifically, we characterize the distribution of the Ridgeless interpolator in high dimensions, in terms of a Ridge estimator in an associated Gaussian sequence model with positive regularization, which plays the role of the prescribed implicit regularization in the context of prediction risk. Our distributional characterizations hold for general random designs and extend uniformly to positively regularized Ridge estimators. As a demonstration of the analytic power of these characterizations, we derive approximate formulae for a general class of weighted $\ell_q$ risks for Ridge(less) estimators that were previously available only for $\ell_2$. Our theory also provides certain further conceptual reconciliation with the conventional wisdom: given any data covariance, a certain amount of regularization in Ridge regression remains beneficial for `most' signals across various statistical tasks including prediction, estimation and inference, as long as the noise level is non-trivial. Surprisingly, optimal tuning can be achieved simultaneously for all the designated statistical tasks by a single generalized or $k$-fold cross-validation scheme, despite being designed specifically for tuning prediction risk.

ON THE SECOND-ORDER INVERSE REGRESSION METHODS FOR A GENERAL TYPE OF ELLIPTICAL PREDICTORS

Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity

Expectile regression for analyzing heteroscedasticity in high dimension

Adjusting Inverse Regression for Predictors with Clustered Distribution

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

Prediction regions through Inverse Regression

High Dimensional Elliptical Sliced Inverse Regression in non-Gaussian Distributions

Improving Dimension Reduction Via Contour-Projection

On the number of variables to use in principal component regression

Generalized principal Hessian directions for mixture multivariate skew elliptical distributions

Regularity of laws via Dirichlet forms: application to quadratic forms in independent and identically distributed random variables

Distribution-Free Robust Linear Regression

High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality

The distribution of Ridgeless least squares interpolators

On Elliptical and Inverse Elliptical Wishart distributions

Functional sufficient dimension reduction through distance covariance

Inverse Moment Methods for Sufficient Forecasting Using High-Dimensional Predictors

Inverse regression for spatially distributed functional data

Large-scale electronic-structure theory and nanoscale defects formed in cleavage process of silicon

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

Variational Inference based on a Subclass of Closed Skew Normals