Abstract:Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family, minimizing a loss that is typically the (reverse) Kullback-Leibler (KL) divergence. Despite its empirical success, the theoretical properties of VI have only received attention recently, and mostly when the parametric family is the one of Gaussians. This work aims to contribute to the theoretical study of VI in the non-Gaussian case by investigating the setting of Mixture of Gaussians with fixed covariance and constant weights. In this view, VI over this specific family can be casted as the minimization of a Mollified relative entropy, i.e. the KL between the convolution (with respect to a Gaussian kernel) of an atomic measure supported on Diracs, and the target distribution. The support of the atomic measure corresponds to the localization of the Gaussian components. Hence, solving variational inference becomes equivalent to optimizing the positions of the Diracs (the particles), which can be done through gradient descent and takes the form of an interacting particle system. We study two sources of error of variational inference in this context when optimizing the mollified relative entropy. The first one is an optimization result, that is a descent lemma establishing that the algorithm decreases the objective at each iteration. The second one is an approximation error, that upper bounds the objective between an optimal finite mixture and the target distribution.

Small-Variance Asymptotics for Dirichlet Process Mixtures of Svms

Infinite SVM: a Dirichlet Process Mixture of Large-margin Kernel Machines.

Scalable Semisupervised GMM for Big Data Quality Prediction in Multimode Processes.

The Infinite Student'S T-Mixture For Robust Modeling

A Novel Scalable Semi-supervised GMM and Its Application for Multimode Process Quality Prediction with Big Data

Stochastic Volatility in Mean: Efficient Analysis by a Generalized Mixture Sampler

Approximate Approach to Train SVM on Very Large Data Sets

A Large Dimensional Analysis of Least Squares Support Vector Machines

Parallel Computing and SGD-Based DPMM for Soft Sensor Development with Large-Scale Semisupervised Data

Sequential Learning for Dirichlet Process Mixtures

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines

DP-space: Bayesian Nonparametric Subspace Clustering with Small-variance Asymptotics.

Infinite Latent SVM for Classification and Multi-task Learning.

Random matrices meet machine learning: a large dimensional analysis of LS-SVM

Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians

New method for solving Ivanov regularization-based support vector machine learning

Learning Infinite-Horizon Average-Reward Linear Mixture MDPs of Bounded Span

SVM Via Saddle Point Optimization: New Bounds and Distributed Algorithms.

Bayesian collapsed Gibbs sampling for a stochastic volatility model with a Dirichlet process mixture

Sampling from Dirichlet process mixture models with unknown concentration parameter: Mixing issues in large data implementations

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models.