Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regression

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum

A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

A Geometrical Analysis of Kernel Ridge Regression and its Applications

Kernel Ridge Regression Inference

Low-rank kernel regression with preserved locality for multi-class analysis

Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning

Universality of kernel random matrices and kernel regression in the quadratic regime

Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators

Kernel Learning in Ridge Regression "Automatically" Yields Exact Low Rank Solution

Asymptotic Theory for Linear Functionals of Kernel Ridge Regression

An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression

Kernel ridge vs. principal component regression: minimax bounds and adaptability of regularization operators

A Duality Analysis of Kernel Ridge Regression in the Noiseless Regime

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

A Bound on the Maximal Marginal Degrees of Freedom

Dimension free ridge regression

On the Saturation Effect of Kernel Ridge Regression

Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration

On the numerical rank of radial basis function kernels in high dimension