Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

Preliminary test almost unbiased ridge estimator in a linear regression model with multivariate Student-t errors

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

The Superiority of Bayes Linear Unbiased Minimum Variance Estimator with Respect to Ridge Estimator

Identifying a class of Ridge-type estimators in binary logistic regression models

The R2 Ridge Trace in 2sls Regression Estimation

Testing covariates in high-dimensional regression

A new ridge type estimator and its performance for the linear regression model: Simulation and application

Stratified incomplete local simplex tests for curvature of nonparametric multiple regression

Dimension free ridge regression

A Joint Test for Parametric Specification and Independence in Nonlinear Regression Models

Exact and efficient multivariate two-sample tests through generalized linear rank statistics

Linear Regression Analysis with Inequality Constraints on the Regression Parameters Via Empirical Likelihood

Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors

The Optimal Property of Almost Unbiased Stein Ridge Type Principal Component Estimator Under Mean Square Error

Robust U-type test for high dimensional regression coefficients using refitted cross-validation variance estimation

Empirical Likelihood Test for Regression Coefficients in High Dimensional Partially Linear Models

A Ridge-Regularized Jackknifed Anderson-Rubin Test

Partial Penalized Likelihood Ratio Test under Sparse Case

Asymptotic Properties of Lasso+mLS and Lasso+Ridge in Sparse High-Dimensional Linear Regression

Towards a theory for testing statistical hypothesis: Multivariate mean with nuisance covariance matrix

Testing Independence Between Exogenous Variables and Unobserved Errors