Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

A Robust Variable Selection Method for Sparse Online Regression Via the Elastic Net Penalty

Penalized Generative Variable Selection

Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

Variable selection for uncertain regression models based on elastic net method

Variable selection and model prediction based on Lasso, adaptive lasso and elastic net

Sparse neural network regression with variable selection

Elastic Net Procedure for Partially Linear Models

Elastic Net with Adaptive Weight for Image Denoising

Variable Selection for Generalized Linear Model with Highly Correlated Covariates

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Robust Variable Selection Based on Relaxed Lad Lasso

A LASSO-Type Approach to Variable Selection and Estimation for Censored Regression Model

Penalized robust estimators in logistic regression with applications to sparse models

Enmsp: an elastic-net multi-step screening procedure for high-dimensional regression

A novel variable selection algorithm for multi-layer perceptron with elastic net.

Stability of the Elastic Net Estimator

Regularization Path For Linear Model Via Net Method

Scaled sparse linear regression

Outlier detection and robust variable selection via the penalized weighted LAD-LASSO method

A Transparent and Nonlinear Method for Variable Selection

Post Selection Shrinkage Estimation for High Dimensional Data Analysis