Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

SCAD-Ridge Penalized Likelihood Estimators for Ultra-high Dimensional Models

Expectile regression for analyzing heteroscedasticity in high dimension

Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity

Penalized Sparse Covariance Regression with High Dimensional Covariates

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Non-convex Penalized Estimation in High-Dimensional Models with Single-Index Structure

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates

Post Selection Shrinkage Estimation for High Dimensional Data Analysis

An RKHS-based approach to double-penalized regression in high-dimensional partially linear models

High-Dimensional Sparse Additive Hazards Regression

High-dimensional variable selection accounting for heterogeneity in regression coefficients across multiple data sources

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Consistent Tuning Parameter Selection in High Dimensional Sparse Linear Regression

A Constructive Approach to High-dimensional Regression

Calibrating nonconvex penalized regression in ultra-high dimension

Simultaneous Estimation and Dataset Selection for Transfer Learning in High Dimensions by a Non-convex Penalty

Robust exponential squared loss-based variable selection for high-dimensional single-index varying-coefficient model

Asymptotic Properties of Lasso+mLS and Lasso+Ridge in Sparse High-Dimensional Linear Regression

Penalized Maximum Tangent Likelihood Estimation and Robust Variable Selection