Abstract:This article considers high-dimensional regression problems in which the number of predictors p exceeds the sample size n. We develop a model-averaging procedure for high-dimensional regression problems. Unlike most variable selection studies featuring the identification of true predictors, our focus here is on the prediction accuracy for the true conditional mean of y given the p predictors. Our method consists of two steps. The first step is to construct a class of regression models, each with a smaller number of regressors, to avoid the degeneracy of the information matrix. The second step is to find suitable model weights for averaging. To minimize the prediction error, we estimate the model weights using a delete-one cross-validation procedure. Departing from the literature of model averaging that requires the weights always sum to one, an important improvement we introduce is to remove this constraint. We derive some theoretical results to justify our procedure. A theorem is proved, showing that delete-one cross-validation achieves the lowest possible prediction loss asymptotically. This optimality result requires a condition that unravels an important feature of high-dimensional regression. The prediction error of any individual model in the class for averaging is required to be higher than the classic root n rate under the traditional parametric regression. This condition reflects the difficulty of high-dimensional regression and it depicts a situation especially meaningful for p > n. We also conduct a simulation study to illustrate the merits of the proposed approach over several existing methods, including lasso, group lasso, forward regression, Phase Coupled (PC)-simple algorithm, Akaike information criterion (AIC) model-averaging, Bayesian information criterion (BIC) model-averaging methods, and SCAD (smoothly clipped absolute deviation). This approach uses quadratic programming to overcome the computing time issue commonly encountered in the cross-validation literature. Supplementary materials for this article are available online.

A general model of regression using iterative series

Recursive Least-Squares Method with Membership Functions

A weighted least squares procedure to approximate least absolute deviation estimation in time series with specific reference to infinite variance unit root problems

Bivariate least squares linear regression: towards a unified analytic formalism

A general formulation of reweighted least squares fitting

Generalized Resubstitution for Regression Error Estimation

An iterative approach for deriving and solving an accurate regression equation

A Model-Averaging Approach for High-Dimensional Regression

Weighted least squares regression with the best robustness and high computability

Extended Linear Regression: A Kalman Filter Approach for Minimizing Loss via Area Under the Curve

Regression and Classification With Spline-Based Separable Expansions

A Novel Theoretical Framework for Exponential Smoothing

Weighted Approximation By Max-Product Generalized Exponential Sampling Series

An Iterative Algorithm for Rescaled Hyperbolic Functions Regression

A Rapidly Convergent Iterative Method for the Solution of the Generalised Nonlinear Least Squares Problem

A Statistical Theory of Regularization-Based Continual Learning

Least Squares Approximation with a Diverging Number of Parameters

A REPRESENTATION FOR THE REGRESSION COEFFICIENTS IN WEIGHTED LEAST SQUARES

Improved Convergence for $\ell_\infty$ and $\ell_1$ Regression via Iteratively Reweighted Least Squares

Iterative algorithms for weighted and unweighted finite-rank time-series approximations

Models as Approximations I: Consequences Illustrated with Linear Regression