Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

On the approximation by weighted ridge functions

K -Functional, Weighted Moduli of Smoothness, and Best Weighted Polynomial Approximation on a Simplex

On best approximation by multivariate ridge functions with applications to generalized translation networks

Approximation of generalized ridge functions in high dimensions

The Matrix Ridge Approximation: Algorithms and Applications

On approximation of functions, conjugate to the functions of several variables belonging to weighted Lipschitz and Zygmund classes

Multivariate approximation of functions on irregular domains by weighted least-squares methods

Approximation error of single hidden layer neural networks with fixed weights

Mean Dimension of Ridge Functions

Approximation in Hilbert spaces of the Gaussian and related analytic kernels

Dimension free ridge regression

Infinite-Variate $L^2$-Approximation with Nested Subspace Sampling

Best approximations for the weighted combination of the Cauchy--Szegö kernel and its derivative in the mean

Dimension Reduction via Gaussian Ridge Functions

Approximation of Bivariate Functions by Generalized Wendland Radial Basis Functions

Linear/Ridge expansions: Enhancing linear approximations by ridge functions

Weighted Least Approximation on Compact Riemannian Manifolds

Convergence and Near-optimal Sampling for Multivariate Function Approximations in Irregular Domains via Vandermonde with Arnoldi

APPLICATION OF L-CURVE METHOD TO EQUIVALENT WEIGHT ROBUST RIDGE ESTIMAION MODEL

Asymptotic Theory for Linear Functionals of Kernel Ridge Regression

An approximation by neural networkswith a fixed weight