Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

Ridge estimation of covariance matrix from data in two classes

Targeted Fused Ridge Estimation of Inverse Covariance Matrices from Multiple High-Dimensional Data Classes

Dimension free ridge regression

Estimation of Error Variance Via Ridge Regression

Covariance-regularized regression and classification for high dimensional problems

On Ridge Estimation in High-dimensional Rotationally Sparse Linear Regression

A semi-automatic method to guide the choice of ridge parameter in ridge regression

SURE Estimates for High Dimensional Classification

Double Data Piling for Heterogeneous Covariance Models

Ridge Fusion in Statistical Learning

Equality between two general ridge estimators and equivalence of their residual sums of squares

Ridgelets: estimating with ridge functions

Random design analysis of ridge regression

The Matrix Ridge Approximation: Algorithms and Applications

Positive Definite Estimation of Large Covariance Matrix Using Generalized Nonconvex Penalties.

Ridge Estimation to the Restricted Linear Model

Risk and cross validation in ridge regression with correlated samples

Discriminative Ridge Machine: A Classifier for High-Dimensional Data or Imbalanced Data

A Mallows-type Model Averaging Estimator for Ridge Regression with Randomly Right Censored Data

Ridge Regularization: an Essential Concept in Data Science

Heritability estimation for a linear combination of phenotypes via ridge regression