Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

Elastic Net Procedure for Partially Linear Models

Dual Lasso Selector

Variable selection for partially linear models via partial correlation

Stability of the Elastic Net Estimator

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

Enmsp: an elastic-net multi-step screening procedure for high-dimensional regression

Regularization Path For Linear Model Via Net Method

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates

Variable selection and model prediction based on Lasso, adaptive lasso and elastic net

Variable Selection for the Partial Linear Single-Index Model

Nonnegative Adaptive Lasso for Ultra-High Dimensional Regression Models and A Two-Stage Method Applied in Financial Modeling

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Consistent group selection in high-dimensional linear regression

Variable Selection for Generalized Linear Model with Highly Correlated Covariates

Variable selection for uncertain regression models based on elastic net method

Feature selection guided by structural information

Variable Selection and Minimax Prediction in High-dimensional Functional Linear Model

Robust Elastic Net Regression

On Model Selection Consistency of the Elastic Net when P >> N

Variable selection for partially linear models via Bayesian subset modeling with diffusing prior

Exploring elastic net and multivariate regression