Abstract:This article considers high-dimensional regression problems in which the number of predictors p exceeds the sample size n. We develop a model-averaging procedure for high-dimensional regression problems. Unlike most variable selection studies featuring the identification of true predictors, our focus here is on the prediction accuracy for the true conditional mean of y given the p predictors. Our method consists of two steps. The first step is to construct a class of regression models, each with a smaller number of regressors, to avoid the degeneracy of the information matrix. The second step is to find suitable model weights for averaging. To minimize the prediction error, we estimate the model weights using a delete-one cross-validation procedure. Departing from the literature of model averaging that requires the weights always sum to one, an important improvement we introduce is to remove this constraint. We derive some theoretical results to justify our procedure. A theorem is proved, showing that delete-one cross-validation achieves the lowest possible prediction loss asymptotically. This optimality result requires a condition that unravels an important feature of high-dimensional regression. The prediction error of any individual model in the class for averaging is required to be higher than the classic root n rate under the traditional parametric regression. This condition reflects the difficulty of high-dimensional regression and it depicts a situation especially meaningful for p > n. We also conduct a simulation study to illustrate the merits of the proposed approach over several existing methods, including lasso, group lasso, forward regression, Phase Coupled (PC)-simple algorithm, Akaike information criterion (AIC) model-averaging, Bayesian information criterion (BIC) model-averaging methods, and SCAD (smoothly clipped absolute deviation). This approach uses quadratic programming to overcome the computing time issue commonly encountered in the cross-validation literature. Supplementary materials for this article are available online.

A Model-averaging Method for High-Dimensional Regression with Missing Responses at Random

A Model-Averaging Approach for High-Dimensional Regression

Ultra-High Dimensional Model Averaging for Multi-Categorical Response

Mean Response Estimation with Missing Response in the Presence of High-Dimensional Covariates

Feature Screening Via Distance Correlation for Ultrahigh Dimensional Data with Responses Missing at Random

Model Averaging Procedure for Varying-Coefficient Partially Linear Models with Missing Responses

Adjusted Feature Screening for Ultra-High Dimensional Missing Response

Optimal model averaging for partially linear models with missing response variables and error‐prone covariates

Jackknife model averaging for linear regression models with missing responses

Rank-Based Greedy Model Averaging for High-Dimensional Survival Data

Jackknife model averaging for high‐dimensional quantile regression

Model Averaging Estimation for Nonparametric Varying-Coefficient Models with Multiplicative Heteroscedasticity

Feature Screening in Ultrahigh-Dimensional Partially Linear Models with Missing Responses at Random

Model Selection and Model Averaging for Semiparametric Partially Linear Models with Missing Data

Dimension reduction with missing response at random

Sequential Model Averaging for High Dimensional Linear Regression Models

Model Averaging for Generalized Linear Model with Covariates that are Missing completely at Random

Focused Vector Information Criterion Model Selection and Model Averaging Regression with Missing Response

Model Averaging Marginal Regression for High Dimensional Conditional Quantile Prediction

Model Averaging for Varying Coefficient Partially Linear Models With Missing Data

Model Averaging with High-Dimensional Dependent Data