Abstract:Trevor Hastie, Robert Tibshirani, Ryan Tibshirani. In exciting recent work, Bertsimas, King and Mazumder ( Ann. Statist. 44 (2016) 813–852) showed that the classical best subset selection problem in regression modeling can be formulated as a mixed integer optimization (MIO) problem. Using recent advances in MIO algorithms, they demonstrated that best subset selection can now be solved at much larger problem sizes than what was thought possible in the statistics community. They presented empirical comparisons of best subset with other popular variable selection procedures, in particular, the lasso and forward stepwise selection. Surprisingly (to us), their simulations suggested that best subset consistently outperformed both methods in terms of prediction accuracy. Here, we present an expanded set of simulations to shed more light on these comparisons. The summary is roughly as follows: •neither best subset nor the lasso uniformly dominate the other, with best subset generally performing better in very high signal-to-noise (SNR) ratio regimes, and the lasso better in low SNR regimes; •for a large proportion of the settings considered, best subset and forward stepwise perform similarly, but in certain cases in the high SNR regime, best subset performs better; •forward stepwise and best subsets tend to yield sparser models (when tuned on a validation set), especially in the high SNR regime; •the relaxed lasso (actually, a simplified version of the original relaxed estimator defined in Meinshausen ( Comput. Statist. Data Anal. 52 (2007) 374–393)) is the overall winner, performing just about as well as the lasso in low SNR scenarios, and nearly as well as best subset in high SNR scenarios.

Best subset, forward stepwise or lasso? Analysis and recommendations based on extensive comparisons

Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons

Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso

Best Subset Selection via a Modern Optimization Lens

Variable Selection with Stepwise and Best Subset Approaches.

COMBSS: best subset selection via continuous optimization

Variable selection in linear regression models: choosing the best subset is not always the best choice

Multi-Model Subset Selection

When do stepwise algorithms meet subset selection criteria?

Dynamic Incremental Optimization for Best Subset Selection

An efficient optimization approach for best subset selection in linear regression, with application to model selection and fitting in autoregressive time-series

Subset selection for multiple linear regression via optimization

A Consistent and Scalable Algorithm for Best Subset Selection in Single Index Models

Adaptive Multiple Comparisons With the Best

Subset Selection by Pareto Optimization

Comprehensive Stepwise Selection for Logistic Regression

Best-Subset Selection in Generalized Linear Models: A Fast and Consistent Algorithm via Splicing Technique

Optimal subgroup selection

Bayesian subset selection and variable importance for interpretable prediction and classification

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

Model selection and estimation in regression with grouped variables