Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

Forward Regression for Ultra-High Dimensional Variable Screening

Greedy Forward Regression for Variable Screening

Profile forward regression screening for ultra-high dimensional semiparametric varying coefficient partially linear models

Faithful Variable Screening for High-Dimensional Convex Regression

Profiled Forward Regression for Ultrahigh Dimensional Variable Screening in Semiparametric Partially Linear Models

Forward Variable Selection for Sparse Ultra-High Dimensional Varying Coefficient Models

Forward variable selection for sparse ultra-high-dimensional generalized varying coefficient models

Ultrahigh dimensional variable selection: beyond the linear model

Bayesian iterative screening in ultra-high dimensional settings

Factor Profiling for Ultra High Dimensional Variable Selection

Strong Sure Screening of Ultra-high Dimensional Categorical Data

Robust variable screening for regression using factor profiling

Feature Screening for High-Dimensional Variable Selection in Generalized Linear Models

A robust variable screening procedure for ultra-high dimensional data

Factor Profiled Sure Independence Screening

Category-Adaptive Variable Screening for Ultra-High Dimensional Heterogeneous Categorical Data

Screen then select: a strategy for correlated predictors in high-dimensional quantile regression

RaSE: A Variable Screening Framework via Random Subspace Ensembles

High-dimensional Ordinary Least-squares Projection for Screening Variables

Nonparametric Independence Screening in Sparse Ultra-High-Dimensional Additive Models

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking