Abstract:Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = \beta^\top x_i +\epsilon_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := \Sigma^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm \Delta)$ for some explicitly small $\Delta$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

Data Clustering Via Uncorrelated Ridge Regression

Robust Subspace Clustering Via Thresholding Ridge Regression

Robust Dual Clustering with Adaptive Manifold Regularization

Exploiting Unsupervised and Supervised Constraints for Subspace Clustering

Kernel two-dimensional ridge regression for subspace clustering

Optimal Margin Distribution Ridge Regression

Co-Clustering With Manifold And Double Sparse Representation

Correspondence-Free Non-Rigid Point Set Registration Using Unsupervised Clustering Analysis

Robust Subspace Learning-Based Low-Rank Representation for Manifold Clustering

Low-rank representation with adaptive dimensionality reduction via manifold optimization for clustering

Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation

Ridge Fuzzy Regression Model

Robust Manifold Matrix Factorization for Joint Clustering and Feature Extraction.

Dimension free ridge regression

Correntropy Induced L2 Graph for Robust Subspace Clustering

Correntropy-Based Low-Rank Matrix Factorization With Constraint Graph Learning for Image Clustering

On Ridge Estimation in High-dimensional Rotationally Sparse Linear Regression

A Flexible Convex Optimization Model for Semi-supervised Clustering with Instance-level Constraints

Supervised Convex Clustering

Unsupervised Manifold Linearizing and Clustering

Low-Rank Tensor Regularized Fuzzy Clustering for Multiview Data