Abstract:Bayesian inference has theoretical attractions as a principled framework for reasoning about beliefs. However, the motivations of Bayesian inference which claim it to be the only 'rational' kind of reasoning do not apply in practice. They create a binary split in which all approximate inference is equally 'irrational'. Instead, we should ask ourselves how to define a spectrum of more- and less-rational reasoning that explains why we might prefer one Bayesian approximation to another. I explore approximate inference in Bayesian neural networks and consider the unintended interactions between the probabilistic model, approximating distribution, optimization algorithm, and dataset. The complexity of these interactions highlights the difficulty of any strategy for evaluating Bayesian approximations which focuses entirely on the method, outside the context of specific datasets and decision-problems. For given applications, the expected utility of the approximate posterior can measure inference quality. To assess a model's ability to incorporate different parts of the Bayesian framework we can identify desirable characteristic behaviours of Bayesian reasoning and pick decision-problems that make heavy use of those behaviours. Here, we use continual learning (testing the ability to update sequentially) and active learning (testing the ability to represent credence). But existing continual and active learning set-ups pose challenges that have nothing to do with posterior quality which can distort their ability to evaluate Bayesian approximations. These unrelated challenges can be removed or reduced, allowing better evaluation of approximate inference methods.

About incoherent inference

Lack of confidence in ABC model choice

Why approximate Bayesian computational (ABC) methods cannot handle model choice problems

A comparison of emulation methods for Approximate Bayesian Computation

An automatic adaptive method to combine summary statistics in approximate Bayesian computation

Generalized Posteriors in Approximate Bayesian Computation

Approximate Bayesian Forecasting

Approximate Bayesian Computation via Classification

Approximate Bayesian Computation for Smoothing

Efficient exact inference for dynamical systems with noisy measurements using sequential approximate Bayesian computation

ABC-CDE: Towards Approximate Bayesian Computation with Complex High-Dimensional Data and Limited Simulations

Summary Statistics in Approximate Bayesian Computation

Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems

Ensemble Kalman inversion approximate Bayesian computation

Accelerating inference for diffusions observed with measurement error and large sample sizes using Approximate Bayesian Computation

Model Misspecification in ABC: Consequences and Diagnostics

Understanding Approximation for Bayesian Inference in Neural Networks

An Approximate Likelihood Perspective on ABC Methods

Monte Carlo algorithms for model assessment via conflicting summaries

Approximate Bayesian Computation for a Class of Time Series Models

Likelihood free inference for Markov processes: a comparison