Abstract:Parallel Markov Chain Monte Carlo (pMCMC) algorithms generate clouds of proposals at each step to efficiently resolve a target probability distribution. We build a rigorous foundational framework for pMCMC algorithms that situates these methods within a unified 'extended phase space' measure-theoretic formalism. Drawing on our recent work that provides a comprehensive theory for reversible single proposal methods, we herein derive general criteria for multiproposal acceptance mechanisms which yield ergodic chains on general state spaces. Our formulation encompasses a variety of methodologies, including proposal cloud resampling and Hamiltonian methods, while providing a basis for the derivation of novel algorithms. In particular, we obtain a top-down picture for a class of methods arising from 'conditionally independent' proposal structures. As an immediate application, we identify several new algorithms including a multiproposal version of the popular preconditioned Crank-Nicolson (pCN) sampler suitable for high- and infinite-dimensional target measures which are absolutely continuous with respect to a Gaussian base measure. To supplement our theoretical results, we carry out a selection of numerical case studies that evaluate the efficacy of these novel algorithms. First, noting that the true potential of pMCMC algorithms arises from their natural parallelizability, we provide a limited parallelization study using TensorFlow and a graphics processing unit to scale pMCMC algorithms that leverage as many as 100k proposals at each step. Second, we use our multiproposal pCN algorithm (mpCN) to resolve a selection of problems in Bayesian statistical inversion for partial differential equations motivated by fluid measurement. These examples provide preliminary evidence of the efficacy of mpCN for high-dimensional target distributions featuring complex geometries and multimodal structures.

Parallel Markov Chain Monte Carlo for Bayesian Hierarchical Models with Big Data, in Two Stages

Parallel Markov Chain Monte Carlo for Non-Gaussian Posterior Distributions

Inferring Hierarchical Mixture Structures: A Bayesian Nonparametric Approach

parallelMCMCcombine: An R Package for Bayesian Methods for Big Data and Analytics

Average of Recentered Parallel MCMC for Big Data

Parallelizing MCMC with Random Partition Trees

Parallel Inference for Big Data with the Group Bayesian Method

Leveraging single-case results to Bayesian hierarchical modelling

Fast maximum likelihood estimation for general hierarchical models

Accelerating Multilevel Markov Chain Monte Carlo Using Machine Learning Models

Parallel MCMC Algorithms: Theoretical Foundations, Algorithm Design, Case Studies

On Markov chain Monte Carlo methods for tall data

Hamiltonian Monte Carlo for Hierarchical Models

Bayesian estimation for longitudinal data in a joint model with HPCs

Parallelization and High-Performance Computing Enables Automated Statistical Inference of Multi-scale Models.

A multiscale Bayesian nonparametric framework for partial hierarchical clustering

Sampling hyperparameters in hierarchical models: improving on Gibbs for high-dimensional latent fields and large data sets

Efficient History Matching Using the Markov-Chain Monte Carlo Method by Means of the Transformed Adaptive Stochastic Collocation Method

Sparse Bayesian Hierarchical Modeling of High-dimensional Clustering Problems

Approximate Gibbs Sampler for Efficient Inference of Hierarchical Bayesian Models for Grouped Count Data

A Deep Learning Method for Comparing Bayesian Hierarchical Models