Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

Consistent Tuning Parameter Selection in High Dimensional Sparse Linear Regression

Consistent Tuning Parameter Selection in High-Dimensional Group-Penalized Regression

Tuning parameter selection in high dimensional penalized likelihood

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Forward Variable Selection for Sparse Ultra-High Dimensional Varying Coefficient Models

Semiparametric Estimation And Variable Selection For Sparse Single Index Models In Increasing Dimension

Shrinkage Tuning Parameter Selection with A Diverging Number of Parameters

Bayesian variable selection in linear regression models with instrumental variables

Variable Selection for High Dimensional Partially Linear Varying Coefficient Errors-in-variables Models

Tuning parameter selector for the penalized likelihood method in multivariate generalized linear models

A Selective Overview of Variable Selection in High Dimensional Feature Space (Invited Review Article)

A High-dimensional M-estimator Framework for Bi-level Variable Selection

Variable Selection in High-Dimensional Quantile Varying Coefficient Models

A MODIFIED BIC TUNING PARAMETER SELECTOR FOR SICA-PENALIZED COX REGRESSION MODELS WITH DIVERGING DIMENSIONALITY

A Simple Information Criterion for Variable Selection in High‐Dimensional Regression

Consistent group selection in high-dimensional linear regression

Variable Selection in Additive Models Via Hierarchical Sparse Penalty

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

Ultrahigh dimensional variable selection: beyond the linear model

Regularization Methods for High-Dimensional Instrumental Variables Regression With an Application to Genetical Genomics

Rates of Convergence of the Adaptive Elastic Net and the Post-Selection Procedure in Ultra-High Dimensional Sparse Models