Abstract:Variational inference with Gaussian mixture models (GMMs) enables learning of highly tractable yet multi-modal approximations of intractable target distributions with up to a few hundred dimensions. The two currently most effective methods for GMM-based variational inference, VIPS and iBayes-GMM, both employ independent natural gradient updates for the individual components and their weights. We show for the first time, that their derived updates are equivalent, although their practical implementations and theoretical guarantees differ. We identify several design choices that distinguish both approaches, namely with respect to sample selection, natural gradient estimation, stepsize adaptation, and whether trust regions are enforced or the number of components adapted. We argue that for both approaches, the quality of the learned approximations can heavily suffer from the respective design choices: By updating the individual components using samples from the mixture model, iBayes-GMM often fails to produce meaningful updates to low-weight components, and by using a zero-order method for estimating the natural gradient, VIPS scales badly to higher-dimensional problems. Furthermore, we show that information-geometric trust-regions (used by VIPS) are effective even when using first-order natural gradient estimates, and often outperform the improved Bayesian learning rule (iBLR) update used by iBayes-GMM. We systematically evaluate the effects of design choices and show that a hybrid approach significantly outperforms both prior works. Along with this work, we publish our highly modular and efficient implementation for natural gradient variational inference with Gaussian mixture models, which supports 432 different combinations of design choices, facilitates the reproduction of all our experiments, and may prove valuable for the practitioner.

Pseudo Independent Conditional Approximation for Training the Mixtures of Gaussian Processes

A unifying view for the mixture model of sparse Gaussian processes

Variational Mixtures of Gaussian Processes for Classification.

An Effective EM Algorithm for Mixtures of Gaussian Processes Via the MCMC Sampling and Approximation.

An Mcmc Based Em Algorithm For Mixtures Of Gaussian Processes

Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models

An Efficient Em Approach To Parameter Learning Of The Mixture Of Gaussian Processes

Variational Bayesian Learning for Parameter Estimation of Mixture of Gaussians

MAP approximation to the variational Bayes Gaussian mixture model and application

Probabilistic Inference from Arbitrary Uncertainty using Mixtures of Factorized Generalized Gaussians

A Gpu-Accelerated Approximate Algorithm for Incremental Learning of Gaussian Mixture Model

A Variational Hardcut EM Algorithm for the Mixtures of Gaussian Processes

Differentiability and Approximation of Probability Functions under Gaussian Mixture Models: A Bayesian Approach

Mixture of robust Gaussian processes and its hard-cut EM algorithm with variational bounding approximation

MCMC methods for Gaussian process models using fast approximations for the likelihood

Pseudo-marginal Bayesian inference for Gaussian process latent variable models

Parameter Estimation of Gaussian Mixture Model Based on Variational Bayesian Learning

A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models

An Effective Model Selection Criterion for Mixtures of Gaussian Processes.

Multiview Learning with Variational Mixtures of Gaussian Processes

Probabilistic Point Set Matching with Gaussian Mixture Model.