Abstract:Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family, minimizing a loss that is typically the (reverse) Kullback-Leibler (KL) divergence. Despite its empirical success, the theoretical properties of VI have only received attention recently, and mostly when the parametric family is the one of Gaussians. This work aims to contribute to the theoretical study of VI in the non-Gaussian case by investigating the setting of Mixture of Gaussians with fixed covariance and constant weights. In this view, VI over this specific family can be casted as the minimization of a Mollified relative entropy, i.e. the KL between the convolution (with respect to a Gaussian kernel) of an atomic measure supported on Diracs, and the target distribution. The support of the atomic measure corresponds to the localization of the Gaussian components. Hence, solving variational inference becomes equivalent to optimizing the positions of the Diracs (the particles), which can be done through gradient descent and takes the form of an interacting particle system. We study two sources of error of variational inference in this context when optimizing the mollified relative entropy. The first one is an optimization result, that is a descent lemma establishing that the algorithm decreases the objective at each iteration. The second one is an approximation error, that upper bounds the objective between an optimal finite mixture and the target distribution.

Stochastic Variational Approach to Minimum Uncertainty States

Variational Inference for Diffusion Processes.

Stochastic viscosity approximations of Hamilton-Jacobi equations and variance reduction

A variational approach to sampling in diffusion processes

Uncertainty Quantification for Markov Processes via Variational Principles and Functional Inequalities

Variational Gaussian Process Diffusion Processes

Variational Inference for Stochastic Differential Equations

Viscous control of minimum uncertainty state in hydrodynamics

A variational approach to nonlinear and interacting diffusions

Stochastic Differential Equations with Variational Wishart Diffusions

Uncertainty Propagation in Stochastic Systems via Mixture Models with Error Quantification

Learning minimal representations of stochastic processes with variational autoencoders

Minimum-uncertainty angular wave packets and quantized mean values

New Minimum Uncertainty States for Thermo Field Dynamics

On Minimum-Dispersion Control of Nonlinear Diffusion Processes

Variational Structures Beyond Gradient Flows: a Macroscopic Fluctuation-Theory Perspective

Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians

A relativistically covariant stochastic model for systems with a fluctuating number of particles

Methods for Solving Variational Inequalities with Markovian Stochasticity

A Viscosity Solution Theory of Stochastic Hamilton-Jacobi-Bellman equations in the Wasserstein Space