Abstract:Over the past decade, characterizing the exact asymptotic risk of regularized estimators in high-dimensional regression has emerged as a popular line of work. This literature considers the proportional asymptotics framework, where the number of features and samples both diverge, at a rate proportional to each other. Substantial work in this area relies on Gaussianity assumptions on the observed covariates. Further, these studies often assume the design entries to be independent and identically distributed. Parallel research investigates the universality of these findings, revealing that results based on the i.i.d.~Gaussian assumption extend to a broad class of designs, such as i.i.d.~sub-Gaussians. However, universality results examining dependent covariates so far focused on correlation-based dependence or a highly structured form of dependence, as permitted by right rotationally invariant designs. In this paper, we break this barrier and study a dependence structure that in general falls outside the purview of these established classes. We seek to pin down the extent to which results based on i.i.d.~Gaussian assumptions persist. We identify a class of designs characterized by a block dependence structure that ensures the universality of i.i.d.~Gaussian-based results. We establish that the optimal values of the regularized empirical risk and the risk associated with convex regularized estimators, such as the Lasso and ridge, converge to the same limit under block dependent designs as they do for i.i.d.~Gaussian entry designs. Our dependence structure differs significantly from correlation-based dependence, and enables, for the first time, asymptotically exact risk characterization in prevalent nonparametric regression problems in high dimensions. Finally, we illustrate through experiments that this universality becomes evident quite early, even for relatively moderate sample sizes.

Universality in block dependent linear models with applications to nonparametric regression

Universality of Estimator for High-Dimensional Linear Models with Block Dependency

Asymptotics of Linear Regression with Linearly Dependent Data

Rates of convergence and normal approximations for estimators of local dependence random graph models

Central Limit Theorems of a Recursive Stochastic Algorithm with Applications to Adaptive Designs

Random Quadratic Forms with Dependence: Applications to Restricted Isometry and Beyond

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

On optimal block resampling for Gaussian-subordinated long-range dependent processes

Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regression

Stability of a Generalized Debiased Lasso with Applications to Resampling-Based Variable Selection

Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance

High Dimensional Logistic Regression Under Network Dependence

Robust Inference for High-dimensional Linear Models with Heavy-tailed Errors via Partial Gini Covariance

Simultaneous support recovery in high dimensions: Benefits and perils of block $\ell_1/\ell_\infty$-regularization

Dimension free ridge regression

High-Dimensional Analysis for Generalized Nonlinear Regression: from Asymptotics to Algorithm

High-dimensional logistic regression with missing data: Imputation, regularization, and universality

Robust Sparse Regression with Non-Isotropic Designs

Uncertainty quantification in high-dimensional linear models incorporating graphical structures with applications to gene set analysis

Finite sample posterior concentration in high-dimensional regression

Distributed non-convex regularization for generalized linear regression