Abstract:Artificial neural networks (ANNs) are powerful machine learning methods used in many modern applications such as facial recognition, machine translation, and cancer diagnostics. A common issue with ANNs is that they usually have millions or billions of trainable parameters, and therefore tend to overfit to the training data. This is especially problematic in applications where it is important to have reliable uncertainty estimates. Bayesian neural networks (BNN) can improve on this, since they incorporate parameter uncertainty. In addition, latent binary Bayesian neural networks (LBBNN) also take into account structural uncertainty by allowing the weights to be turned on or off, enabling inference in the joint space of weights and structures. In this paper, we will consider two extensions to the LBBNN method: Firstly, by using the local reparametrization trick (LRT) to sample the hidden units directly, we get a more computationally efficient algorithm. More importantly, by using normalizing flows on the variational posterior distribution of the LBBNN parameters, the network learns a more flexible variational posterior distribution than the mean field Gaussian. Experimental results show that this improves predictive power compared to the LBBNN method, while also obtaining more sparse networks. We perform two simulation studies. In the first study, we consider variable selection in a logistic regression setting, where the more flexible variational distribution leads to improved results. In the second study, we compare predictive uncertainty based on data generated from two-dimensional Gaussian distributions. Here, we argue that our Bayesian methods lead to more realistic estimates of predictive uncertainty.

A Note on Bayesian Networks with Latent Root Variables

Bayesian model selection for the latent position cluster model for Social Networks

Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables

Learning a Bayesian network with multiple latent variables for implicit relation representation

Sparsifying Bayesian neural networks with latent binary variables and normalizing flows

A Latent-Variable Bayesian Nonparametric Regression Model

Bayesian networks and probabilistic reasoning about scientific evidence when there is a lack of data

Empirical Bayes in Bayesian learning: understanding a common practice

Grand canonical ensembles of sparse networks and Bayesian inference

Normalized Latent Measure Factor Models

Bayesian Approach to Linear Bayesian Networks

A Latent Space Theory for Emergent Abilities in Large Language Models

The Bridged Posterior: Optimization, Profile Likelihood and a New Approach to Generalized Bayes

Belief Update in CLG Bayesian Networks With Lazy Propagation

PAC-Bayesian Theory Meets Bayesian Inference

On Theory for BART

Discovering Causes and Effects of a Given Node in Bayesian Networks

Latent Agents in Networks: Estimation and Targeting

Higher-Order Bayesian Networks, Exactly (Extended version)

Nonparametric Bayesian networks are typically faithful in the total variation metric

Reliable and Efficient Inference of Bayesian Networks from Sparse Data by Statistical Learning Theory