Abstract:Over the last two decades, many exciting variable selection methods have been developed for finding a small group of covariates that are associated with the response from a large pool. Can the discoveries by such data mining approaches be spurious due to high dimensionality and limited sample size? Can our fundamental assumptions on exogeneity of covariates needed for such variable selection be validated with the data? To answer these questions, we need to derive the distributions of the maximum spurious correlations given certain number of predictors, namely, the distribution of the correlation of a response variable Y with the best s linear combinations of p covariates X, even when X and Y are independent. When the covariance matrix of X possesses the restricted eigenvalue property, we derive such distributions for both finite s and diverging s, using Gaussian approximation and empirical process techniques. However, such a distribution depends on the unknown covariance matrix of X. Hence, we use the multiplier bootstrap procedure to approximate the unknown distributions and establish the consistency of such a simple bootstrap approach. The results are further extended to the situation where residuals are from regularized fits. Our approach is then applied to construct the upper confidence limit for the maximum spurious correlation and testing exogeneity of covariates. The former provides a baseline for guarding against false discoveries due to data mining and the latter tests whether our fundamental assumptions for high-dimensional model selection are statistically valid. Our techniques and results are illustrated by both numerical examples and real data analysis.

On Distribution-Weighted Partial Least Squares with Diverging Number of Highly Correlated Predictors

Estimation and model selection in generalized additive partial linear models for correlated data with diverging number of covariates

Penalized least squares estimation with weakly dependent data

Penalized Independence Rule for Testing High-Dimensional Hypotheses

Iteratively Reweighted Least Squares Method for Estimating Polyserial and Polychoric Correlation Coefficients

Stable Direction Recovery in Single-Index Models with a Diverging Number of Predictors

Doubly Distributed Supervised Learning and Inference with High-Dimensional Correlated Outcomes

A distribution-based LASSO for a general single-index model

ASYMPTOTIC DISTRIBUTIONS OF HIGH-DIMENSIONAL DISTANCE CORRELATION INFERENCE

Conditional Uncorrelation and Efficient Subset Selection in Sparse Regression

Generalized Regression Estimators with High-Dimensional Covariates

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

ARE DISCOVERIES SPURIOUS? DISTRIBUTIONS OF MAXIMUM SPURIOUS CORRELATIONS AND THEIR APPLICATIONS

ON THE SECOND-ORDER INVERSE REGRESSION METHODS FOR A GENERAL TYPE OF ELLIPTICAL PREDICTORS

Adaptive and Reversed Penalty for Analysis of High-Dimensional Correlated Data

Adjusting Inverse Regression for Predictors with Clustered Distribution

Communication-efficient distributed estimator for generalized linear models with a diverging number of covariates

Distributionally weighted least squares in structural equation modeling.

Dimension Reduction Estimation for Central Mean Subspace with Missing Multivariate Response.

Predictive Distributions and the Transition from Sparse to Dense Functional Data

Combined-Penalized Likelihood Estimations with A Diverging Number of Parameters