Abstract:SIAM/ASA Journal on Uncertainty Quantification, Volume 12, Issue 1, Page 69-100, March 2024. Variational inference is an approximation framework for Bayesian inference that seeks to improve quantified uncertainty in predictions by optimizing a simplified distribution over parameters to stand in for the full posterior. Capturing model variations that remain consistent with training data enables more robust predictions by reducing parameter sensitivity. This work introduces a fixed-point optimization for variational inference that is applicable when every feasible log density can be expressed as a linear combination of functions from a given basis. In such cases, the optimizer becomes a fixed-point of projective integral updates. When the basis spans univariate quadratics in each parameter, the feasible distributions are Gaussian mean-fields and the projective integral updates yield quasi-Newton variational Bayes (QNVB). Other bases and updates are also possible. Since these updates require high-dimensional integration, this work begins by proposing an efficient quasirandom sequence of quadratures for mean-field distributions. Each iterate of the sequence contains two evaluation points that combine to correctly integrate all univariate quadratic functions and, if the mean-field factors are symmetric, all univariate cubics. More importantly, averaging results over short subsequences achieves periodic exactness on a much larger space of multivariate polynomials of quadratic total degree. The corresponding variational updates require four loss evaluations with standard (not second-order) backpropagation to eliminate error terms from over half of all multivariate quadratic basis functions. This integration technique is motivated by first proposing stochastic blocked mean-field quadratures, which may be useful in other contexts. A PyTorch implementation of QNVB allows for better control over model uncertainty during training than competing methods. Experiments demonstrate superior generalizability for multiple learning problems and architectures.

Efficient quadratures for high-dimensional Bayesian data assimilation

Conditional Bayesian Quadrature

Adaptive sampling-based quadrature rules for efficient Bayesian prediction

Projective Integral Updates for High-Dimensional Variational Inference

Exploiting gradients and Hessians in Bayesian optimization and Bayesian quadrature

Quantum Approximate Optimization Algorithm and Quantum-enhanced Markov Chain Monte Carlo: A Hybrid Approach to Data Assimilation in 4DVAR

Enhancing Quasi-Monte Carlo Methods by Exploiting Additive Approximation for Problems in Finance

Fast Automatic Bayesian Cubature Using Matching Kernels and Designs

ABC-CDE: Towards Approximate Bayesian Computation with Complex High-Dimensional Data and Limited Simulations

Parallel adaptive Bayesian quadrature for rare event estimation

Bayesian Numerical Integration with Neural Networks

Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature

Parallelization of adaptive Bayesian cubature using multimodal optimization algorithms

High-dimensional copula variational approximation through transformation

Exact and optimal quadratization of nonlinear finite-dimensional non-autonomous dynamical systems

Quasi-Monte Carlo for Bayesian design of experiment problems governed by parametric PDEs

A Hybrid Monte-Carlo Sampling Smoother for Four Dimensional Data Assimilation

Efficient, multimodal, and derivative-free Bayesian inference with Fisher-Rao gradient flows

SOBER: Highly Parallel Bayesian Optimization and Bayesian Quadrature over Discrete and Mixed Spaces

A note on the accuracy of adaptive Gauss–Hermite quadrature