Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

Local Influence Analysis for The Ridge Regression Under Stochastic Linear Restrictions

Local Linear M‐estimation in Non‐parametric Spatial Regression

A new stochastic mixed ridge estimator in linear regression model

A New Ridge-Type Estimator in Stochastic Restricted Linear Regression

Ridge Estimation to the Restricted Linear Model

Assessing local influence for elliptical linear models under equality constraints

High-leverage Measures in Linear Regression Diagnostics

Local influence analysis for penalized Gaussian likelihood estimation in partially linear single-index models

Local Influence Analysis for Penalized Gaussian Likelihood Estimators in Partially Linear Models

On the Stochastic Restricted Almost Unbiased Estimators in Linear Regression Model

Local influence analysis for Poisson autoregression with an application to stock transaction data

A Stochastic Restricted Ridge Regression Estimator in Partially Linear Varying Coefficient Models

Positive-rule Stein-Type Almost Unbiased Ridge Estimator in Linear Regression Model

On Ridge Estimation in High-dimensional Rotationally Sparse Linear Regression

On the Restricted Almost Unbiased Estimators in Linear Regression

Dimension free ridge regression

Preliminary test almost unbiased ridge estimator in a linear regression model with multivariate Student-t errors

The R2 Ridge Trace in 2sls Regression Estimation

On the Weighted Mixed Almost Unbiased Ridge Estimator in Stochastic Restricted Linear Regression

Local Linear Estimation for Regression Models with Locally Stationary Long Memory Errors

Local Influence for Semiparametric Generalized Linear Model