Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

Rows vs. Columns: Randomized Kaczmarz or Gauss-Seidel for Ridge Regression

A Kaczmarz Method with Simple Random Sampling for Solving Large Linear Systems

Convergence properties of the randomized extended Gauss-Seidel and Kaczmarz methods

Randomized symmetric Gauss-Seidel method for solving linear least squares problems

A Note on the Randomized Kaczmarz Algorithm for Solving Doubly Noisy Linear Systems

Randomized Kaczmarz with tail averaging

Dimension free ridge regression

A Note on Randomized Kaczmarz Algorithm for Solving Doubly-Noisy Linear Systems

Regularized randomized iterative algorithms for factorized linear systems

Randomized block Kaczmarz method with projection for solving least squares

Randomized Kaczmarz Algorithm for Inconsistent Linear Systems: An Exact MSE Analysis

Randomized Extended Kaczmarz for Solving Least-Squares

Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration

Iterative methods for solving factorized linear systems

An Asynchronous Parallel Randomized Kaczmarz Algorithm

On Convergence Rates of Kaczmarz-Type Methods with Different Selection Rules of Working Rows

Acceleration of Randomized Kaczmarz Method via the Johnson-Lindenstrauss Lemma

Nonlinear Kaczmarz algorithms and their convergence

An accelerated randomized Kaczmarz algorithm

Rates of convergence of randomized Kaczmarz algorithms in Hilbert spaces

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems