Abstract:Large scale Bayesian nonparametrics (BNP) learner such as stochastic variational inference (SVI) can handle datasets with large class number and large training size at fractional cost. Like its predecessor, SVI rely on the assumption of conjugate variational posterior to approximate the true posterior. A more challenging problem is to consider large scale learning on non-conjugate posterior. Recent works in this direction are mostly associated with using Monte Carlo methods for approximating the learner. However, these works are usually demonstrated on non-BNP related task and less complex models such as logistic regression, due to higher computational complexity. In order to overcome the issue faced by SVI, we develop a novel approach based on the recently proposed variational maximization-maximization (VMM) learner to allow large scale learning on non-conjugate posterior. Unlike SVI, our VMM learner does not require closed-form expression for the variational posterior expectatations. Our only requirement is that the variational posterior is differentiable. In order to ensure convergence in stochastic settings, SVI rely on decaying step-sizes to slow its learning. Inspired by SVI and Adam, we propose the novel use of decaying step-sizes on both gradient and ascent direction in our VMM to significantly improve its learning. We show that our proposed methods is compatible with ResNet features when applied to large class number datasets such as MIT67 and SUN397. Finally, we compare our proposed learner with several recent works such as deep clustering algorithms and showed we were able to produce on par or outperform the state-of-the-art methods in terms of clustering measures.

Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models

On the Convergence of Coordinate Ascent Variational Inference

Understanding Stochastic Natural Gradient Variational Inference

Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

Automated Augmented Conjugate Inference for Non-conjugate Gaussian Process Models

Robust Coordinate Ascent Variational Inference with Markov chain Monte Carlo simulations

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

Adaptive variational Bayes: Optimality, computation and applications

Monte Carlo Co-Ordinate Ascent Variational Inference

Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

A Deterministic Global Optimization Method for Variational Inference

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Using Large Ensembles of Control Variates for Variational Inference

Stochastic Learning of Non-Conjugate Variational Posterior for Image Classification

Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models

Mean-field Variational Inference via Wasserstein Gradient Flow

Convergence of variational Monte Carlo simulation and scale-invariant pre-training

On the Convergence of Black-Box Variational Inference

Variational Inference with Coverage Guarantees in Simulation-Based Inference

High-dimensional copula variational approximation through transformation

Convergence of coordinate ascent variational inference for log-concave measures via optimal transport