Abstract:It has been observed that the performances of many high-dimensional estimation problems are universal with respect to underlying sensing (or design) matrices. Specifically, matrices with markedly different constructions seem to achieve identical performance if they share the same spectral distribution and have ``generic'' singular vectors. We prove this universality phenomenon for the case of convex regularized least squares (RLS) estimators under a linear regression model with additive Gaussian noise. Our main contributions are two-fold: (1) We introduce a notion of universality classes for sensing matrices, defined through a set of deterministic conditions that fix the spectrum of the sensing matrix and precisely capture the previously heuristic notion of generic singular vectors; (2) We show that for all sensing matrices that lie in the same universality class, the dynamics of the proximal gradient descent algorithm for solving the regression problem, as well as the performance of RLS estimators themselves (under additional strong convexity conditions) are asymptotically identical. In addition to including i.i.d. Gaussian and rotational invariant matrices as special cases, our universality class also contains highly structured, strongly correlated, or even (nearly) deterministic matrices. Examples of the latter include randomly signed versions of incoherent tight frames and randomly subsampled Hadamard transforms. As a consequence of this universality principle, the asymptotic performance of regularized linear regression on many structured matrices constructed with limited randomness can be characterized by using the rotationally invariant ensemble as an equivalent yet mathematically more tractable surrogate.

Some Overall Properties of Seemingly Unrelated Regression Models

Seemingly unrelated nonparametric models with positive correlation and constrained error variances

Robust Inference for Seemingly Unrelated Regression Models

Varying Coefficient Partially Nonlinear Models with Nonstationary Regressors

Statistical inference on seemingly unrelated non-parametric regression models with serially correlated errors

ROBUST ESTIMATION IN PARAMETRIC TIME SERIES MODELS UNDER LONG‐ AND SHORT‐RANGE‐DEPENDENT STRUCTURES

Asymptotic Behavior for S-estimators in Random Design Linear Model with Long-Range-dependent Errors

Large System Of Seemingly Unrelated Regressions: A Penalized Quasi-Maximum Likelihood Estimation Perspective

Statistical Inference of Seemingly Unrelated Partially Linear Additive Regression Models

General Expression of Best Equivariant Estimators of Regression Coefficients in Seemingly Unrelated Regression Models

Models as Approximations I: Consequences Illustrated with Linear Regression

Efficient Estimation for Error Component Seemingly Unrelated Nonparametric Regression Models

With random regressors, least squares inference is robust to correlated errors with unknown correlation structure

Seemingly unrelated regression with measurement error: estimation via Markov Chain Monte Carlo and mean field variational Bayes approximation

Seemingly unrelated penalized regression models

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

Using mixtures in seemingly unrelated linear regression models with non-normal errors

Asymptotic Properties of Marginal Least-Square Estimator for Ultrahigh-Dimensional Linear Regression Models with Correlated Errors

Low-rank matrix estimation via nonconvex spectral regularized methods in errors-in-variables matrix regression

Families of Parsimonious Finite Mixtures of Regression Models