Abstract:Artificial neural networks (ANNs) are powerful machine learning methods used in many modern applications such as facial recognition, machine translation, and cancer diagnostics. A common issue with ANNs is that they usually have millions or billions of trainable parameters, and therefore tend to overfit to the training data. This is especially problematic in applications where it is important to have reliable uncertainty estimates. Bayesian neural networks (BNN) can improve on this, since they incorporate parameter uncertainty. In addition, latent binary Bayesian neural networks (LBBNN) also take into account structural uncertainty by allowing the weights to be turned on or off, enabling inference in the joint space of weights and structures. In this paper, we will consider two extensions to the LBBNN method: Firstly, by using the local reparametrization trick (LRT) to sample the hidden units directly, we get a more computationally efficient algorithm. More importantly, by using normalizing flows on the variational posterior distribution of the LBBNN parameters, the network learns a more flexible variational posterior distribution than the mean field Gaussian. Experimental results show that this improves predictive power compared to the LBBNN method, while also obtaining more sparse networks. We perform two simulation studies. In the first study, we consider variable selection in a logistic regression setting, where the more flexible variational distribution leads to improved results. In the second study, we compare predictive uncertainty based on data generated from two-dimensional Gaussian distributions. Here, we argue that our Bayesian methods lead to more realistic estimates of predictive uncertainty.

Sparse Bayesian Linear Regression Using Generalized Normal Priors.

Sparse estimation of linear model via Bayesian method

On high-dimensional classification by sparse generalized Bayesian logistic regression

Neuronized Priors for Bayesian Sparse Linear Regression

Empirical Bayes inference in sparse high-dimensional generalized linear models

Bayesian High-dimensional Linear Regression with Sparse Projection-posterior

Tail-adaptive Bayesian shrinkage

On The Sparse Bayesian Learning Of Linear Models

Probabilistic Classifiers with a Generalized Gaussian Scale Mixture Prior

Sparse bayesian inference with regularized gaussian distributions

Bayesian Joint Spike-and-Slab Graphical Lasso

Flexible Bayesian quantile regression for nonlinear mixed effects models based on the generalized asymmetric Laplace distribution

Log-regularly varying scale mixture of normals for robust regression

Bayesian shrinkage prediction for the regression problem

Bayesian Analysis for Over-parameterized Linear Model without Sparsity

EP-GIG Priors and Applications in Bayesian Sparse Learning

Hierarchical sparsity priors for regression models

Sparsifying Bayesian neural networks with latent binary variables and normalizing flows

Dynamic sparsity on dynamic regression models

High-dimensional Grouped-regression using Bayesian Sparse Projection-posterior