Abstract:In this paper, we study efficient approximate sampling for probability distributions known up to normalization constants. We specifically focus on a problem class arising in Bayesian inference for large-scale inverse problems in science and engineering applications. The computational challenges we address with the proposed methodology are: (i) the need for repeated evaluations of expensive forward models; (ii) the potential existence of multiple modes; and (iii) the fact that gradient of, or adjoint solver for, the forward model might not be feasible. While existing Bayesian inference methods meet some of these challenges individually, we propose a framework that tackles all three systematically. Our approach builds upon the Fisher-Rao gradient flow in probability space, yielding a dynamical system for probability densities that converges towards the target distribution at a uniform exponential rate. This rapid convergence is advantageous for the computational burden outlined in (i). We apply Gaussian mixture approximations with operator splitting techniques to simulate the flow numerically; the resulting approximation can capture multiple modes thus addressing (ii). Furthermore, we employ the Kalman methodology to facilitate a derivative-free update of these Gaussian components and their respective weights, addressing the issue in (iii). The proposed methodology results in an efficient derivative-free posterior approximation method, flexible enough to handle multi-modal distributions: Gaussian Mixture Kalman Inversion (GMKI). The effectiveness of GMKI is demonstrated both theoretically and numerically in several experiments with multimodal target distributions, including proof-of-concept and two-dimensional examples, as well as a large-scale application: recovering the Navier-Stokes initial condition from solution data at positive times.

Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations

Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models

A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models

Mean-field Variational Inference via Wasserstein Gradient Flow

Fast approximation of variational bayes dirichlet process mixture using the maximization–maximization algorithm

Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models

A generalized mean field algorithm for variational inference in exponential families

Understanding Stochastic Natural Gradient Variational Inference

Gradient-free variational learning with conditional mixture networks

Learning Gaussian Mixtures Using the Wasserstein-Fisher-Rao Gradient Flow

Accelerated Stochastic Variational Inference

Natural Gradient Variational Bayes Without Fisher Matrix Analytic Calculation and Its Inversion

Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation

Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

MAP approximation to the variational Bayes Gaussian mixture model and application

Noisy Natural Gradient As Variational Inference

Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians

Efficient, multimodal, and derivative-free Bayesian inference with Fisher-Rao gradient flows

Variational inference for Dirichlet process mixtures

Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference