Stability and L2-penalty in Model Averaging

Hengkun Zhu,Guohua Zou

2023-11-23

Abstract:Model averaging has received much attention in the past two decades, which integrates available information by averaging over potential models. Although various model averaging methods have been developed, there are few literatures on the theoretical properties of model averaging from the perspective of stability, and the majority of these methods constrain model weights to a simplex. The aim of this paper is to introduce stability from statistical learning theory into model averaging. Thus, we define the stability, asymptotic empirical risk minimizer, generalization, and consistency of model averaging and study the relationship among them. Our results indicate that stability can ensure that model averaging has good generalization performance and consistency under reasonable conditions, where consistency means model averaging estimator can asymptotically minimize the mean squared prediction error. We also propose a L2-penalty model averaging method without limiting model weights and prove that it has stability and consistency. In order to reduce the impact of tuning parameter selection, we use 10-fold cross-validation to select a candidate set of tuning parameters and perform a weighted average of the estimators of model weights based on estimation errors. The Monte Carlo simulation and an illustrative application demonstrate the usefulness of the proposed method.

Machine Learning

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Introducing the concept of stability**: The paper attempts to introduce the concept of "stability" from statistical learning theory into model averaging. Although there are various model averaging methods, theoretical research on the stability of model averaging is limited. The authors define the stability, asymptotic empirical risk minimization (AERM), generalization ability, and consistency of model averaging, and explore the relationships among them. The research results show that under reasonable conditions, stability can ensure good generalization performance and consistency of model averaging.

2. **Proposing an L2 penalty method for unconstrained weights**: For cases where model weights are not restricted, the paper proposes a new L2-penalty model averaging method, which does not constrain model weights and demonstrates its stability and consistency. To reduce the impact of parameter selection, the authors use 10-fold cross-validation to select a set of candidate parameters and perform weighted averaging based on estimation errors.

Through the above two aspects of research, the paper aims to address the shortcomings of model averaging in stability theory and propose an improved method to enhance the generalization ability of model averaging.