Abstract:We propose the Bayesian bridge estimator for regularized regression and classification. Two key mixture representations for the Bayesian bridge model are developed: (1) a scale mixture of normals with respect to an alpha-stable random variable; and (2) a mixture of Bartlett--Fejer kernels (or triangle densities) with respect to a two-component mixture of gamma random variables. Both lead to MCMC methods for posterior simulation, and these methods turn out to have complementary domains of maximum efficiency. The first representation is a well known result due to West (1987), and is the better choice for collinear design matrices. The second representation is new, and is more efficient for orthogonal problems, largely because it avoids the need to deal with exponentially tilted stable random variables. It also provides insight into the multimodality of the joint posterior distribution, a feature of the bridge model that is notably absent under ridge or lasso-type priors. We prove a theorem that extends this representation to a wider class of densities representable as scale mixtures of betas, and provide an explicit inversion formula for the mixing distribution. The connections with slice sampling and scale mixtures of normals are explored. On the practical side, we find that the Bayesian bridge model outperforms its classical cousin in estimation and prediction across a variety of data sets, both simulated and real. We also show that the MCMC for fitting the bridge model exhibits excellent mixing properties, particularly for the global scale parameter. This makes for a favorable contrast with analogous MCMC algorithms for other sparse Bayesian models. All methods described in this paper are implemented in the R package BayesBridge. An extensive set of simulation results are provided in two supplemental files.

Multi-Response Bridge Regularization Parameter Selection via Multivariate Generalized Information Criterion

Selection of tuning parameters in bridge regression models via Bayesian information criterion

A Generalized Focused Information Criterion for GMM

The Bayesian Bridge

The Best Fit Bayesian Hierarchical Generalized Linear Model Selection Using Information Complexity Criteria in the MCMC Approach

Adaptive bridge regression modeling with model selection criteria

Gibbs-Based Information Criteria and the Over-Parameterized Regime

Large-Scale Model Selection with Misspecification

Statistical Inference for Generalized Additive Models: Simultaneous Confidence Corridors and Variable Selection

Variable Selection for High-dimensional Generalized Linear Models using an Iterated Conditional Modes/Medians Algorithm

Information based model selection criteria for generalized linear mixed models with unknown variance component parameters

Decompositions of Information Criteria in Joint Model Under Probit Link for Response

An Effective Model Selection Criterion for Mixtures of Gaussian Processes.

A MODIFIED BIC TUNING PARAMETER SELECTOR FOR SICA-PENALIZED COX REGRESSION MODELS WITH DIVERGING DIMENSIONALITY

Generalized covariance‐based inference for models set‐identified from independence restrictions

Multimodel Inference Based on Smoothed Information Criteria

Empirical-likelihood-based criteria for model selection on marginal analysis of longitudinal data with dropout missingness

Automated Model Selection for Generalized Linear Models

Consistent Tuning Parameter Selection in High-Dimensional Group-Penalized Regression

Multi-Response Heteroscedastic Gaussian Process Models and Their Inference

Variable Selection in Generalized Estimating Equations Via Empirical Likelihood and Gaussian Pseudo-Likelihood