Abstract:Variational inference (VI) is a widely used framework in Bayesian estimation. For most of the non-Gaussian statistical models, it is infeasible to find an analytically tractable solution to estimate the posterior distributions of the parameters. Recently, an improved framework, namely the extended variational inference (EVI), has been introduced and applied to derive analytically tractable solution by employing lower-bound approximation to the variational objective function. Two conditions required for EVI implementation, namely the weak condition and the strong condition, are discussed and compared in this paper. In practical implementation, the convergence of the EVI depends on the selection of the lower-bound approximation, no matter with the weak condition or the strong condition. In general, two approximation strategies, the single lower-bound (SLB) approximation and the multiple lower-bounds (MLB) approximation, can be applied to carry out the lower-bound approximation. To clarify the differences between the SLB and the MLB, we will also discuss the convergence properties of the aforementioned two approximations. Extensive comparisons are made based on some existing EVI-based non-Gaussian statistical models. Theoretical analysis are conducted to demonstrate the differences between the weak and the strong conditions. Qualitative and quantitative experimental results are presented to show the advantages of the SLB approximation.

Derivations of Variational Gaussian Process Approximation Framework

Variational Gaussian Processes: A Functional Analysis View

Amortized Variational Inference for Deep Gaussian Processes

DGP-LVM: Derivative Gaussian process latent variable models

Variational Gaussian Process Diffusion Processes

Recyclable Gaussian Processes

A New Variational Radial Basis Function Approximation for Inference in Multivariate Diffusions

Variational Inference for Uncertainty on the Inputs of Gaussian Process Models

Sparse Orthogonal Variational Inference for Gaussian Processes

Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation

Variational Inference for Diffusion Processes.

Variational methods for the Dirichlet process

Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

Estimation of Dynamic Gaussian Processes

Wasserstein Gradient Flow over Variational Parameter Space for Variational Inference

Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows

Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models

Variational Bayesian Inference for Partially Observed Diusions

Variational Inference for Gaussian Process Modulated Poisson Processes

Stein Variational Gaussian Processes

On the Convergence of Extended Variational Inference for Non-Gaussian Statistical Models.