Abstract:The fit of a regression model to new data is often worse due to overfitting. Analysts use variable selection techniques to develop parsimonious regression models, which may introduce bias into regression estimates. Shrinkage methods have been proposed to mitigate overfitting and reduce bias in estimates. Post‐estimation shrinkage is an alternative to penalized methods. This study evaluates effectiveness of post‐estimation shrinkage in improving prediction performance of full and selected models. Through a simulation study, results were compared with ordinary least squares (OLS) and ridge in full models, and best subset selection (BSS) and lasso in selected models. We focused on prediction errors and the number of selected variables. Additionally, we proposed a modified version of the parameter‐wise shrinkage (PWS) approach named non‐negative PWS (NPWS) to address weaknesses of PWS. Results showed that no method was superior in all scenarios. In full models, NPWS outperformed global shrinkage, whereas PWS was inferior to OLS. In low correlation with moderate‐to‐high signal‐to‐noise ratio (SNR), NPWS outperformed ridge, but ridge performed best in small sample sizes, high correlation, and low SNR. In selected models, all post‐estimation shrinkage performed similarly, with global shrinkage slightly inferior. Lasso outperformed BSS and post‐estimation shrinkage in small sample sizes, low SNR, and high correlation but was inferior when the opposite was true. Our study suggests that, with sufficient information, NPWS is more effective than global shrinkage in improving prediction accuracy of models. However, in high correlation, small sample sizes, and low SNR, penalized methods generally outperform post‐estimation shrinkage methods.

Shrinkage Tuning Parameter Selection with A Diverging Number of Parameters

Variable Selection for Generalized Varying Coefficient Partially Linear Models with Diverging Number of Parameters

Tuning Parameter Selectors for the Smoothly Clipped Absolute Deviation Method.

Regularization and variable selection for infinite variance autoregressive models

Tuning parameter selection in high dimensional penalized likelihood

A MODIFIED BIC TUNING PARAMETER SELECTOR FOR SICA-PENALIZED COX REGRESSION MODELS WITH DIVERGING DIMENSIONALITY

Least Squares Approximation with a Diverging Number of Parameters

Tuning parameter selection in econometrics

Shrinkage-based regularization tests for high-dimensional data with application to gene set analysis

Shrinkage Estimation of the Varying Coefficient Model

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

Tuning Parameter Selection in Adaptive LASSO for Quantile Regression with Penal Data

Shrinkage Methods for Treatment Choice

Information criteria for structured parameter selection in high dimensional tree and graph models

Multivariate Regression Shrinkage and Selection by Canonical Correlation Analysis

Regression Coefficient and Autoregressive Order Shrinkage and Selection Via the Lasso

Adaptive Bi-Level Variable Selection for Quantile Regression Models with a Diverging Number of Covariates

Variable selection for semiparametric varying-coefficient spatial autoregressive models with a diverging number of parameters

Post‐Estimation Shrinkage in Full and Selected Linear Regression Models in Low‐Dimensional Data Revisited

Tuning parameter selection for penalized likelihood estimation of inverse covariance matrix

A framework to select tuning parameters for nonparametric derivative estimation