Abstract:Abstract Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper, we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2300 data-generating scenarios, including both synthetic and semisynthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a “no panacea” view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.

Exploring Dimension Learning Via a Penalized Probabilistic Principal Component Analysis

Dynamic Principal Component Analysis in High Dimensions

Penalized Independence Rule for Testing High-Dimensional Hypotheses

Joint Principal Component and Discriminant Analysis for Dimensionality Reduction

Supervised dimensionality reduction for big data

Principal component analysis balancing prediction and approximation accuracy for spatial data

Distance Metric Learning with Penalized Linear Discriminant Analysis

Prescriptive PCA: Dimensionality Reduction for Two-stage Stochastic Optimization

Sparse Unsupervised Dimensionality Reduction Algorithms

Latent Variable Models for Dimensionality Reduction

Dynamic Principal Subspaces in High Dimensions

Maximally Correlated Principal Component Analysis Based on Deep Parameterization Learning.

Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery

An Experimental Study of Dimension Reduction Methods on Machine Learning Algorithms with Applications to Psychometrics

Dimension reduction via score ratio matching

Bayesian Maximum Margin Principal Component Analysis

SCAD-Ridge Penalized Likelihood Estimators for Ultra-high Dimensional Models

A Hybrid PCA-LDA Model for Dimension Reduction.

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Dimension Reduction Based on Sampling

Penalized minimum average variance estimation