Abstract:Bayesian inference provides a methodology for parameter estimation and uncertainty quantification in machine learning and deep learning methods. Variational inference and Markov Chain Monte-Carlo (MCMC) sampling methods are used to implement Bayesian inference. In the past few decades, MCMC sampling methods have faced challenges in being adapted to larger models (such as deep learning models) and big data problems. Advanced proposal distributions that incorporate gradients, such as a Langevin proposal distribution, provide a means to address some of the limitations of MCMC sampling for Bayesian neural networks. Furthermore, MCMC methods have typically been constrained to statisticians, and hence not well-known among deep learning researchers. We present a tutorial for MCMC methods that covers simple Bayesian linear and logistic models, and Bayesian neural networks. The aim of this tutorial is to bridge the gap between theory and implementation via Python code, given a general sparsity of libraries and tutorials. This tutorial provides code in Python with data and instructions that enable their use and extension. We provide results for selected benchmark problems showing the strengths and weaknesses of implementing the respective Bayesian models via MCMC. We highlight the challenges in sampling multi-modal posterior distributions for the case of Bayesian neural networks and the need for further improvement of convergence diagnosis methods.

A tutorial on Bayesian nonparametric models

Bayesian Hypothesis Tests Using Nonparametric Statistics

Revisiting k-means: New Algorithms via Bayesian Nonparametrics

Bayesian Nonparametrics: An Alternative to Deep Learning

Bayesian Neural Networks via MCMC: A Python-Based Tutorial

Bayesian mixtures of common factor analyzers: Model, variational inference, and applications

Non-parametric Bayesian modeling of complex networks

Nonparametric Bayesian Methods (Dirichlet Process Mixtures)

Bayesian Latent Class Analysis Tutorial

Bayesian Model Selection in High-Dimensional Settings

Bayesian Nonparametric Covariance Regression

Bayesian Nonparametric Variable Selection as an Exploratory Tool for Finding Genes that Matter

Bayesian inference: More than Bayes's theorem

A Kernel Approach to Tractable Bayesian Nonparametrics

Optimal Bayesian estimators for latent variable cluster models

A Tutorial on Bayesian Optimization

Practical Bayesian Inference in Neuroscience: Or How I Learned To Stop Worrying and Embrace the Distribution

A Tutorial on Parametric Variational Inference

A Conceptual Introduction to Bayesian Model Averaging

Some Case Studies Using Bayesian Statistical Models