Abstract:Abstract Markov chain Monte Carlo (MCMC)-based simulation approaches are by far the most common method in Bayesian inference to access the posterior distribution. Recently, motivated by successes in machine learning, variational inference (VI) has gained in interest in statistics since it promises a computationally efficient alternative to MCMC enabling approximate access to the posterior. Classical approaches such as mean-field VI (MFVI), however, are based on the strong mean-field assumption for the approximate posterior where parameters or parameter blocks are assumed to be mutually independent. As a consequence, parameter uncertainties are often underestimated and alternatives such as semi-implicit VI (SIVI) have been suggested to avoid the mean-field assumption and to improve uncertainty estimates. SIVI uses a hierarchical construction of the variational parameters to restore parameter dependencies and relies on a highly flexible implicit mixing distribution whose probability density function is not analytic but samples can be taken via a stochastic procedure. With this paper, we investigate how different forms of VI perform in semiparametric additive regression models as one of the most important fields of application of Bayesian inference in statistics. A particular focus is on the ability of the rivalling approaches to quantify uncertainty, especially with correlated covariates that are likely to aggravate the difficulties of simplifying VI assumptions. Moreover, we propose a method, where we combine both advantages of MFVI and SIVI and compare its performance. The different VI approaches are studied in comparison with MCMC in simulations and an application to tree height models of douglas fir based on a large-scale forestry data set.

Markovian Score Climbing: Variational Inference with KL(p||q)

Transport Score Climbing: Variational Inference Using Forward KL and Adaptive Neural Transport

Convergence of the Inexact Langevin Algorithm and Score-based Generative Models in KL Divergence

Score-Based Variational Inference for Inverse Problems

Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference

Variational (Gradient) Estimate Of The Score Function In Energy-Based Latent Variable Models

Variational Inference: A Review for Statisticians

Monte Carlo Co-Ordinate Ascent Variational Inference

Variational Hamiltonian Monte Carlo via Score Matching

Batch, match, and patch: low-rank approximations for score-based variational inference

Robust Coordinate Ascent Variational Inference with Markov chain Monte Carlo simulations

Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence

Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix

Importance Sampled Stochastic Optimization for Variational Inference

Variational inference: uncertainty quantification in additive models

Accelerated Stochastic Variational Inference

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Batch and match: black-box variational inference with a score-based divergence

Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models

On the Convergence of Coordinate Ascent Variational Inference

Semi-Implicit Variational Inference via Score Matching