Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

A Group Adaptive Elastic-Net Approach for Variable Selection in High-Dimensional Linear Regression

Nonnegative Estimation and Variable Selection Via Adaptive Elastic-Net for High-Dimensional Data

Adaptive Lasso in high-dimensional settings

Elastic Net Procedure for Partially Linear Models

Consistent group selection in high-dimensional linear regression

Rates of Convergence of the Adaptive Elastic Net and the Post-Selection Procedure in Ultra-High Dimensional Sparse Models

Adaptive Elastic Net and Separate Selection from Least Squares for Ultra-High Dimensional Regression Models

The Application and Property of Elastic Net Procedure for Partially Linear Models

Variable Selection and Collinearity Processing for Multivariate Data Via Row-Elastic-net Regularization

Grouping Variable Selection by Weight Fused Elastic Net for Multi-Collinear Data

A Robust Variable Selection Method for Sparse Online Regression Via the Elastic Net Penalty

Nonnegative Adaptive Lasso for Ultra-High Dimensional Regression Models and A Two-Stage Method Applied in Financial Modeling

Group Variable Selection for Data with Dependent Structures

Bi-level variable selection via adaptive sparse group Lasso

A Link-Free Sparse Group Variable Selection Method for Single-Index Model

A Two-Stage Variable Selection Approach for Correlated High Dimensional Predictors

Versatile Descent Algorithms for Group Regularization and Variable Selection in Generalized Linear Models

Consistent Tuning Parameter Selection in High Dimensional Sparse Linear Regression

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Multi-step Adaptive Elastic-Net: Reducing False Positives in High-Dimensional Variable Selection

Variable Selection and Minimax Prediction in High-dimensional Functional Linear Model