Abstract:Expectation propagation (EP) is a widely successful way to approximate the posteriors of complex Bayesian models. However, it suffers from expensive memory and time overheads, since it involves local approximations with locally specific messages. A recent art, namely averaged EP (AEP), upgrades EP by leveraging the average message effect on the posterior distribution, instead of the locally specific ones, so as to simultaneously reduce memory and time costs. In this paper, we extend AEP to a novel black-box expectation propagation (abbr. BBEP) algorithm, which can be directly applied to many Bayesian models without model-specific derivations. We leverage three ideas of black-box learning, leading to three versions of BBEP, referred to as BBEPm, BBEPg and BBEPo with Monte Carlo moment matching, Monte Carlo gradients and objective of AEP, respectively. For variance reduction, the importance sampling is used, and the proposal distribution selection as well as high dimensionality setting is discussed. Furthermore, we develop online versions of BBEP for optimization speedup given large-scale data sets. We empirically compare BBEP against the state-of-the-art black-box baseline algorithms on both synthetic and real-world data sets. Experimental results demonstrate that BBEP outperforms the baseline algorithms and it is even on a par with analytical solutions in some settings.

Expectation Propogation for approximate inference in dynamic Bayesian networks

Expectation Propagation for approximate Bayesian inference

An Expectation Propagation Perspective on Approximate Message Passing

Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference

Probabilistic Inference of Bayesian Neural Networks with Generalized Expectation Propagation

Loopy belief propagation for approximate inference: An empirical study

Approximate inference for dynamic Bayesian networks: sliding window approach

Double-Chain Unscented Expectation Propagation for Inference in Stochastic Dynamical Models of Biological Processes.

Approximate Inference and Constrained Optimization

Circular Belief Propagation for Approximate Probabilistic Inference

Approximate Posterior Inference for Bayesian Models: Black-Box Expectation Propagation

Understanding belief propagation and its generalizations

Approximating the Permanent with Belief Propagation

Generalizing Expectation Propagation with Mixtures of Exponential Family Distributions and an Application to Bayesian Logistic Regression.

Expectation propagation for continuous time stochastic processes

Expectation Propagation in the large-data limit

Approximate evaluation of marginal association probabilities with belief propagation

Expectation Propagation with Stochastic Kinetic Model in Complex Interaction Systems.

Expectation propagation for Poisson data

Message Passing with L1 Penalized KL Minimization

Fixing the Bethe approximation: How structural modifications in a graph improve belief propagation