Abstract:For most of the non-Gaussian statistical models, the data being modeled represent strongly structured properties, such as scalar data with bounded support (e.g., beta distribution), vector data with unit length (e.g., Dirichlet distribution), and vector data with positive elements (e.g., generalized inverted Dirichlet distribution). In practical implementations of non-Gaussian statistical models, it is infeasible to find an analytically tractable solution to estimating the posterior distributions of the parameters. Variational inference (VI) is a widely used framework in Bayesian estimation. Recently, an improved framework, namely, the extended VI (EVI), has been introduced and applied successfully to a number of non-Gaussian statistical models. EVI derives analytically tractable solutions by introducing lower bound approximations to the variational objective function. In this paper, we compare two approximation strategies, namely, the multiple lower bounds (MLBs) approximation and the single lower bound (SLB) approximation, which can be applied to carry out the EVI. For implementation, two different conditions, the weak and the strong conditions, are discussed. Convergence of the EVI depends on the selection of the lower bound, regardless of the choice of weak or strong condition. We also discuss the convergence properties to clarify the differences between MLB and SLB. Extensive comparisons are made based on some EVI-based non-Gaussian statistical models. Theoretical analysis is conducted to demonstrate the differences between the weak and strong conditions. Experimental results based on real data show advantages of the SLB approximation over the MLB approximation.

Statistical Guarantees and Algorithmic Convergence Issues of Variational Boosting

On the Convergence of Extended Variational Inference for Non-Gaussian Statistical Models.

Theoretical Convergence Guarantees for Variational Autoencoders

A Deterministic Global Optimization Method for Variational Inference

$α$-Variational Inference with Statistical Guarantees

Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians

Insights into Multiple/Single Lower Bound Approximation for Extended Variational Inference in Non-Gaussian Structured Data Modeling

Variational Inference: A Review for Statisticians

On the Convergence of Black-Box Variational Inference

Provable convergence guarantees for black-box variational inference

Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference

Stochastic Methods in Variational Inequalities: Ergodicity, Bias and Refinements

Statistical guarantees for stochastic Metropolis-Hastings

On the properties of variational approximations of Gibbs posteriors

Using Large Ensembles of Control Variates for Variational Inference

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Variational Bayesian inference with stochastic search

Optimization by gradient boosting

Optimistic Dual Extrapolation for Coherent Non-monotone Variational Inequalities

On the Convergence of Coordinate Ascent Variational Inference

A Novel Family of Boosted Online Regression Algorithms with Strong Theoretical Bounds