Abstract:Model averaging (MA), a technique for combining estimators from a set of candidate models, has attracted increasing attention in machine learning and statistics. In the existing literature, there is an implicit understanding that MA can be viewed as a form of shrinkage estimation that draws the response vector towards the subspaces spanned by the candidate models. This paper explores this perspective by establishing connections between MA and shrinkage in a linear regression setting with multiple nested models. We first demonstrate that the optimal MA estimator is the best linear estimator with monotonically non-increasing weights in a Gaussian sequence model. The Mallows MA (MMA), which estimates weights by minimizing the Mallows' $C_p$ over the unit simplex, can be viewed as a variation of the sum of a set of positive-part Stein estimators. Indeed, the latter estimator differs from the MMA only in that its optimization of Mallows' $C_p$ is within a suitably relaxed weight set. Motivated by these connections, we develop a novel MA procedure based on a blockwise Stein estimation. The resulting Stein-type MA estimator is asymptotically optimal across a broad parameter space when the variance is known. Numerical results support our theoretical findings. The connections established in this paper may open up new avenues for investigating MA from different perspectives. A discussion on some topics for future research concludes the paper.

Jackknife Model Averaging for Quantile Single-Index Coefficient Model

Jackknife Model Averaging for Mixed-Data Kernel-Weighted Spline Quantile Regressions

Jackknife Model Averaging for Composite Quantile Regression

Jackknife Model Averaging for Additive Expectile Prediction

Model averaging by jackknife criterion in models with dependent data

Institutional Knowledge at Singapore Management University Institutional Knowledge at Singapore Management University Jackknife Model Averaging for Quantile Regressions Jackknife Model Averaging for Quantile Regressions

Model Averaging Estimation for Nonparametric Varying-Coefficient Models with Multiplicative Heteroscedasticity

Jackknife model averaging for linear regression models with missing responses

A Model Averaging Approach for the Ordered Probit and Nested Logit Models with Applications

Penalized Time-Varying Model Averaging

INFERENCE AFTER MODEL AVERAGING IN LINEAR REGRESSION MODELS

Model averaging in a multiplicative heteroscedastic model

Model Averaging for Estimating Treatment Effects With Binary Responses

Model averaging: A shrinkage perspective

A Scalable Frequentist Model Averaging Method

Model Averaging Based on Generalized Method of Moments

Model Averaging for Accelerated Failure Time Models with Missing Censoring Indicators

Time-varying model averaging

Optimal Model Averaging for Divergent-Dimensional Poisson Regressions

Model averaging for right censored data with measurement error

Optimal averaging for functional linear quantile regression models