Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

Application of Random Matrix Theory in High-Dimensional Statistics

Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications

Identity Tests for High Dimensional Data Using Rmt

An Introduction to Complex Random Tensors

Cleaning large correlation matrices: Tools from Random Matrix Theory

High dimensional correlation matrices: the central limit theorem and its applications

High dimensional statistical inference: theoretical development to data analytics

Random Matrix Theory in Lattice Statistical Mechanics

Riemannian statistics meets random matrix theory: towards learning from high-dimensional covariance matrices

Non-asymptotic theory of random matrices: extreme singular values

Spectral properties of high dimensional rescaled sample correlation matrices

Nonlinear Random Matrices and Applications to the Sum of Squares Hierarchy

Framework of Random Matrix Theory for Power System Data Mining in a Non-Gaussian Environment.

Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond

Lectures on random matrix models. The Riemann-Hilbert approach

High-dimensional Factor Model and Its Applications to Statistical Machine Learning

Dimension free ridge regression

Tensor Methods in High Dimensional Data Analysis: Opportunities and Challenges

A Simple Introduction to Free Probability Theory and its Application to Random Matrices

Random matrices applied to anti-symmetric operators

Spatial-Temporal Data Analysis in Nonlinear System