Abstract:We investigate the high-dimensional linear regression problem in the presence of noise correlated with Gaussian covariates. This correlation, known as endogeneity in regression models, often arises from unobserved variables and other factors. It has been a major challenge in causal inference and econometrics. When the covariates are high-dimensional, it has been common to assume sparsity on the true parameters and estimate them using regularization, even with the endogeneity. However, when sparsity does not hold, it has not been well understood to control the endogeneity and high dimensionality simultaneously. This study demonstrates that an estimator without regularization can achieve consistency, that is, benign overfitting, under certain assumptions on the covariance matrix. Specifically, our results show that the error of this estimator converges to zero when the covariance matrices of correlated noise and instrumental variables satisfy a condition on their eigenvalues. We consider several extensions relaxing these conditions and conduct experiments to support our theoretical findings. As a technical contribution, we utilize the convex Gaussian minimax theorem (CGMT) in our dual problem and extend CGMT itself.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to study the issue of overfitting in high-dimensional linear regression models when the noise is correlated with the covariates. Specifically, the authors focus on estimating parameters using the instrumental variable framework in high-dimensional non-sparse parameter settings and demonstrate that an unregularized (i.e., non-ridge type) estimator can achieve consistent estimation under certain conditions. #### Main Issues: 1. **Correlation between noise and covariates**: The paper explores how to perform effective estimation in high-dimensional linear regression when the noise is correlated with the covariates. This correlation is commonly referred to as endogeneity, which is a significant challenge in causal inference and econometrics. 2. **Non-sparse parameters**: The paper assumes that the true parameters are not sparse, meaning that most coordinates are non-zero. In this context, it investigates how to control for endogeneity and the simultaneous impact of high dimensionality. #### Research Contributions: 1. **Theoretical Analysis**: The paper demonstrates that under certain conditions on the covariance matrix, an unregularized estimator can achieve consistent estimation, i.e., benign overfitting. 2. **Condition Analysis**: The paper provides sufficient conditions on the data distribution that ensure the estimation error converges to zero. These conditions involve the effective rank of the covariance matrix of the instrumental variables. 3. **Technical Contributions**: The paper utilizes the Convex Gaussian Minimax Theorem (CGMT) to prove its results and extends the CGMT itself to accommodate the case of non-orthogonal covariance matrices. Through this research, the paper shows that in high-dimensional and non-sparse parameter settings, using the instrumental variable framework can effectively address the correlation between noise and covariates and achieve consistent estimation.

Benign Overfitting of Non-Sparse High-Dimensional Linear Regression with Correlated Noise

Sparse Linear Regression when Noises and Covariates are Heavy-Tailed and Contaminated by Outliers

Outlier Robust and Sparse Estimation of Linear Regression Coefficients

Bayesian variable selection in linear regression models with instrumental variables

Penalized Sparse Covariance Regression with High Dimensional Covariates

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

Bayesian Analysis for Over-parameterized Linear Model without Sparsity

Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension

Conditional Sparse Linear Regression

Testing a Single Regression Coefficient in High Dimensional Regression Model

Bayesian High-dimensional Linear Regression with Sparse Projection-posterior

Sparse Data-Driven Random Projection in Regression for High-Dimensional Data

Testing a Single Regression Coefficient in High Dimensional Linear Models

Asymptotics of Linear Regression with Linearly Dependent Data

Greedy Sensor Selection for Weighted Linear Least Squares Estimation Under Correlated Noise

Asymptotic Properties of Marginal Least-Square Estimator for Ultrahigh-Dimensional Linear Regression Models with Correlated Errors

Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regression

Principal regression for high dimensional covariance matrices

On the effect of noise on fitting linear regression models

Least-Squares Independence Regression for Non-Linear Causal Inference under Non-Gaussian Noise

Sparse Reduced Rank Huber Regression in High Dimensions