Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

A ridge estimation method for the Waring regression model: simulation and application

New ridge parameter estimators for the quasi-Poisson ridge regression model

A new ridge type estimator and its performance for the linear regression model: Simulation and application

New ridge estimators in the inverse Gaussian regression: Monte Carlo simulation and application to chemical data

New ridge parameter estimators for the zero-inflated Conway Maxwell Poisson ridge regression model

Modified ridge-type for the Poisson regression model: simulation and application

Mitigating Multicollinearity in Regression: A Study on Improved Ridge Estimators

Restricted Ridge Regression estimator as a parameter estimation in multiple linear regression model for multicollinearity case

Weighted penalized m-estimators in robust ridge regression: an application to gasoline consumption data

Ridge-Type Pretest and Shrinkage Estimation Strategies in Spatial Error Models with an Application to a Real Data Example

Modified ridge estimator in the Bell regression model

The R2 Ridge Trace in 2sls Regression Estimation

APPLICATION OF L-CURVE METHOD TO EQUIVALENT WEIGHT ROBUST RIDGE ESTIMAION MODEL

A Stochastic Restricted Ridge Regression Estimator in Partially Linear Varying Coefficient Models

Adaptive Ridge Approach to Heteroscedastic Regression

Liu-type estimator in Conway–Maxwell–Poisson regression model: theory, simulation and application

Ridge estimation for uncertain autoregressive moving average model with imprecise observations

EM algorithm for generalized Ridge regression with spatial covariates

Nonlinear Ridge Regression Modeling Method Based on Radial Basis Function and Its Simulation Research

Adaptive Ridge Regression System for Software Cost Estimating on Multi-Collinear Datasets

Dimension free ridge regression