Abstract:We propose the Bayesian bridge estimator for regularized regression and classification. Two key mixture representations for the Bayesian bridge model are developed: (1) a scale mixture of normals with respect to an alpha-stable random variable; and (2) a mixture of Bartlett--Fejer kernels (or triangle densities) with respect to a two-component mixture of gamma random variables. Both lead to MCMC methods for posterior simulation, and these methods turn out to have complementary domains of maximum efficiency. The first representation is a well known result due to West (1987), and is the better choice for collinear design matrices. The second representation is new, and is more efficient for orthogonal problems, largely because it avoids the need to deal with exponentially tilted stable random variables. It also provides insight into the multimodality of the joint posterior distribution, a feature of the bridge model that is notably absent under ridge or lasso-type priors. We prove a theorem that extends this representation to a wider class of densities representable as scale mixtures of betas, and provide an explicit inversion formula for the mixing distribution. The connections with slice sampling and scale mixtures of normals are explored. On the practical side, we find that the Bayesian bridge model outperforms its classical cousin in estimation and prediction across a variety of data sets, both simulated and real. We also show that the MCMC for fitting the bridge model exhibits excellent mixing properties, particularly for the global scale parameter. This makes for a favorable contrast with analogous MCMC algorithms for other sparse Bayesian models. All methods described in this paper are implemented in the R package BayesBridge. An extensive set of simulation results are provided in two supplemental files.

Adaptive group bridge estimation for high-dimensional partially linear models

Robust group non-convex estimations for high-dimensional partially linear models

Adaptive Bi-Level Variable Selection for Quantile Regression Models with a Diverging Number of Covariates

Consistent group selection in high-dimensional linear regression

A High-dimensional M-estimator Framework for Bi-level Variable Selection

A Lower Bound Based Smoothed Quasi-Newton Algorithm for Group Bridge Penalized Regression.

Statistical Limits of Adaptive Linear Models: Low-Dimensional Estimation and Inference

Adaptive bridge regression modeling with model selection criteria

Specification Testing of Partially Linear Single-Index Models: a Groupwise Dimension Reduction-Based Adaptive-to-model Approach

Estimation and model selection in generalized additive partial linear models for correlated data with diverging number of covariates

Which bridge estimator is optimal for variable selection?

Covariate-adjusted response-adaptive designs for generalized linear models

Estimation of A Groupwise Additive Multiple-Index Model and Its Applications

Robust and sparse estimators for linear regression models

The Bayesian Bridge

A novel Bayesian computational approach for bridge-randomized quantile regression in high dimensional models

Doubly robust and efficient estimators for heteroscedastic partially linear single-index models allowing high dimensional covariates

Distributed debiased estimation of high-dimensional partially linear models with jumps

Group Regularized Estimation under Structural Hierarchy

Efficient Estimation for Longitudinal Networks via Adaptive Merging

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates