Abstract:Subset selection is a valuable tool for interpretable learning, scientific discovery, and data compression. However, classical subset selection is often avoided due to selection instability, lack of regularization, and difficulties with post-selection inference. We address these challenges from a Bayesian perspective. Given any Bayesian predictive model $\mathcal{M}$, we extract a family of near-optimal subsets of variables for linear prediction or classification. This strategy deemphasizes the role of a single "best" subset and instead advances the broader perspective that often many subsets are highly competitive. The acceptable family of subsets offers a new pathway for model interpretation and is neatly summarized by key members such as the smallest acceptable subset, along with new (co-) variable importance metrics based on whether variables (co-) appear in all, some, or no acceptable subsets. More broadly, we apply Bayesian decision analysis to derive the optimal linear coefficients for any subset of variables. These coefficients inherit both regularization and predictive uncertainty quantification via $\mathcal{M}$. For both simulated and real data, the proposed approach exhibits better prediction, interval estimation, and variable selection than competing Bayesian and frequentist selection methods. These tools are applied to a large education dataset with highly correlated covariates. Our analysis provides unique insights into the combination of environmental, socioeconomic, and demographic factors that predict educational outcomes, and identifies over 200 distinct subsets of variables that offer near-optimal out-of-sample predictive accuracy.

Economic variable selection

Bayesian variable selection in linear regression models with instrumental variables

Variable Selection in Macroeconomic Forecasting with Many Predictors

Examining variable selection methods for the predictive performance of regression models and the proportion of selected variables and selected random variables

Model selection and estimation in regression with grouped variables

Comprehensive Stepwise Selection for Logistic Regression

Bayesian subset selection and variable importance for interpretable prediction and classification

Variable Selection with Stepwise and Best Subset Approaches.

Bayesian Variable Selection with Related Predictors

Comparison of Bayesian predictive methods for model selection

Variable Selection for Spatial Autoregressive Models with a Diverging Number of Parameters

Model-independent variable selection via the rule-based variable priority

Macroeconomic Forecasting and Variable Selection with a Very Large Number of Predictors: A Penalized Regression Approach

Variable Selection in High-dimensional Data: Method Comparison and its Application in Tax Assessment

Variable selection for uncertain regression models based on elastic net method

Robust Variable Selection and Estimation Based on Kernel Modal Regression

Cost-efficient Variable Selection Using Branching LARS

Factor models and variable selection in high-dimensional regression analysis

Some Aspects of Response Variable Selection and Estimation in Multivariate Linear Regression

Application of Bayesian variable selection in logistic regression model

Nonparametric Assessment of Variable Selection and Ranking Algorithms