Abstract:Variable selection in linear regression settings is a much discussed problem. Best subset selection (BSS) is often considered the intuitive 'gold standard', with its use being restricted only by its NP-hard nature. Alternatives such as the least absolute shrinkage and selection operator (Lasso) or the elastic net (Enet) have become methods of choice in high-dimensional settings. A recent proposal represents BSS as a mixed integer optimization problem so that much larger problems have become feasible in reasonable computation time. We present an extensive neutral comparison assessing the variable selection performance, in linear regressions, of BSS compared to forward stepwise selection (FSS), Lasso and Enet. The simulation study considers a wide range of settings that are challenging with regard to dimensionality (with respect to the number of observations and variables), signal-to-noise ratios and correlations between predictors. As main measure of performance, we used the best possible F1-score for each method to ensure a fair comparison irrespective of any criterion for choosing the tuning parameters, and results were confirmed by alternative performance measures. Somewhat surprisingly, it was only in settings where the signal-to-noise ratio was high and the variables were (nearly) uncorrelated that BSS reliably outperformed the other methods, even in low-dimensional settings. Further, the FSS's performance was nearly identical to BSS. Our results shed new light on the usual presumption of BSS being, in principle, the best choice for variable selection. Especially for correlated variables, alternatives like Enet are faster and appear to perform better in practical settings.

A review of variable selection methods in Partial Least Squares Regression

A review of variable selection methods in Partial Least Squares Regression

Comparison of variable selection methods in partial least squares regression

Variable Selection for Generalized Varying Coefficient Partially Linear Models with Diverging Number of Parameters

Variable Selection in Discriminant Partial Least-Squares Analysis

A Selective Overview of Variable Selection in High Dimensional Feature Space (Invited Review Article)

Comprehensive Stepwise Selection for Logistic Regression

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

Comparison of variable selection methods for PLS-based soft sensor modeling

Examining variable selection methods for the predictive performance of regression models and the proportion of selected variables and selected random variables

Variable selection for both outcomes and predictors: sparse multivariate principal covariates regression

Variable selection in linear regression models: choosing the best subset is not always the best choice

Variable Selection with Stepwise and Best Subset Approaches.

Variable selection using axis-aligned random projections for partial least-squares regression

Nonparametric Assessment of Variable Selection and Ranking Algorithms

Model selection for partial least squares regression

Sparse partial least squares regression for simultaneous dimension reduction and variable selection

Variable Selection for Linear Mixed Effects Model Via Penalization Approaches

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates

Variable selection for partially linear models via Bayesian subset modeling with diffusing prior