Abstract:This article considers high-dimensional regression problems in which the number of predictors p exceeds the sample size n. We develop a model-averaging procedure for high-dimensional regression problems. Unlike most variable selection studies featuring the identification of true predictors, our focus here is on the prediction accuracy for the true conditional mean of y given the p predictors. Our method consists of two steps. The first step is to construct a class of regression models, each with a smaller number of regressors, to avoid the degeneracy of the information matrix. The second step is to find suitable model weights for averaging. To minimize the prediction error, we estimate the model weights using a delete-one cross-validation procedure. Departing from the literature of model averaging that requires the weights always sum to one, an important improvement we introduce is to remove this constraint. We derive some theoretical results to justify our procedure. A theorem is proved, showing that delete-one cross-validation achieves the lowest possible prediction loss asymptotically. This optimality result requires a condition that unravels an important feature of high-dimensional regression. The prediction error of any individual model in the class for averaging is required to be higher than the classic root n rate under the traditional parametric regression. This condition reflects the difficulty of high-dimensional regression and it depicts a situation especially meaningful for p > n. We also conduct a simulation study to illustrate the merits of the proposed approach over several existing methods, including lasso, group lasso, forward regression, Phase Coupled (PC)-simple algorithm, Akaike information criterion (AIC) model-averaging, Bayesian information criterion (BIC) model-averaging methods, and SCAD (smoothly clipped absolute deviation). This approach uses quadratic programming to overcome the computing time issue commonly encountered in the cross-validation literature. Supplementary materials for this article are available online.

Averaged Chain Substitution Method – Applicability, Advantages, and Disadvantages

A Testing Procedure for Determining the Number of Factors in Approximate Factor Models With Large Datasets

Additive Structural Decomposition Analysis and Index Number Theory: An Empirical Application of the Montgomery Decomposition

Identifying Observed Factors in Approximate Factor Models: Estimation and Hypothesis Testing

Comparisons of Four Methods for Estimating a Dynamic Factor Model

A Combined Approach to the Inference of Conditional Factor Models

A factor-adjusted multiple testing of general alternatives

A Multi-Step Procedure to Determine the Number of Factors in Large Approximate Factor Models

Transformed contribution ratio test for the number of factors in static approximate factor models

Knowing Factors or Factor Loadings, or Neither? Evaluating Estimators of Large Covariance Matrices with Noisy and Asynchronous Data

Confirmatory factor analysis: Model testing of financial ratio's with decision support systems approach

A Comparison of Global Factor Models

Factor-adjusted Multiple Testing of Correlations

Beyond step-down analysis: A new test for decomposing the importance of dependent variables in MANOVA.

A Model-Averaging Approach for High-Dimensional Regression

Mining Big Data Using Parsimonious Factor, Machine Learning, Variable Selection and Shrinkage Methods

Factor Models for Asset Returns Based on Transformed Factors

Simultaneous Inference For High-Dimensional Approximate Factor Model

Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution

A Performance Study of Data Mining Techniques: Multiple Linear Regression vs. Factor Analysis

Factor Tree Copula Models for Item Response Data