Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

SCAD-Penalized Least Absolute Deviation Regression in High-Dimensional Models

A new penalized least absolute deviation model for high dimensional sparse linear regression and an efficient sequential linear programming algorithm

Adaptive Lasso in high-dimensional settings

SCAD Penalized Rank Regression with a Diverging Number of Parameters.

Penalized LAD regression for single index models

SCAD‐penalized Quantile Regression for High‐dimensional Data Analysis and Variable Selection

SCAD-Ridge Penalized Likelihood Estimators for Ultra-high Dimensional Models

Quadratic Approximation via the SCAD Penalty with a Diverging Number of Parameters

Penalized and Constrained LAD Estimation in Fixed and High Dimension

VARIABLE SELECTION FOR COVARIATE ADJUSTED REGRESSION MODEL

Robust Regression Shrinkage and Consistent Variable Selection Through the LAD-Lasso

Fused LASSO Penalized Least Absolute Deviation Estimator for High Dimensional Linear Regression

Regularized Estimation for the Least Absolute Relative Error Models with a Diverging Number of Covariates.

Robust Variable Selection Based on Relaxed Lad Lasso

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

The adaptive L1-penalized LAD regression for partially linear single-index models

Non-convex Penalized Estimation in High-Dimensional Models with Single-Index Structure

Group Variable Selection Via Scad-L-2

Outlier detection and robust variable selection via the penalized weighted LAD-LASSO method

A descent algorithm for constrained LAD-Lasso estimation with applications in portfolio selection

Least Squares Approximation with a Diverging Number of Parameters