Abstract:Existing Bayesian models, especially nonparametric Bayesian methods, rely on specially conceived priors to incorporate domain knowledge for discovering improved latent representations. While priors can affect posterior distributions through Bayes' rule, imposing posterior regularization is arguably more direct and in some cases more natural and general. In this paper, we present regularized Bayesian inference (RegBayes), a novel computational framework that performs posterior inference with a regularization term on the desired post-data posterior distribution under an information theoretical formulation. RegBayes is more flexible than the procedure that elicits expert knowledge via priors, and it covers both directed Bayesian networks and undirected Markov networks whose Bayesian formulation results in hybrid chain graph models. When the regularization is induced from a linear operator on the posterior distributions, such as the expectation operator, we present a general convex-analysis theorem to characterize the solution of RegBayes. Furthermore, we present two concrete examples of RegBayes, infinite latent support vector machines (iLSVM) and multi-task infinite latent support vector machines (MT-iLSVM), which explore the large-margin idea in combination with a nonparametric Bayesian model for discovering predictive latent features for classification and multi-task learning, respectively. We present efficient inference methods and report empirical studies on several benchmark datasets, which appear to demonstrate the merits inherited from both large-margin learning and Bayesian nonparametrics. Such results were not available until now, and contribute to push forward the interface between these two important subfields, which have been largely treated as isolated in the community.

Learning Harmonium Models with Infinite Latent Features

Infinite Exponential Family Harmoniums

MMH: Maximum Margin Supervised Harmoniums

Infinite Latent SVM for Classification and Multi-task Learning.

Discriminative infinite latent feature models

Bayesian Active Learning for Discrete Latent Variable Models

Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs

Learning Hierarchical Features with Joint Latent Space Energy-Based Prior

Discriminative Nonparametric Latent Feature Relational Models with Data Augmentation

Max-Margin Nonparametric Latent Feature Models for Link Prediction.

Symphony in the Latent Space: Provably Integrating High-dimensional Techniques with Non-linear Machine Learning Models

Learning Multi-Layer Latent Variable Model with Short Run MCMC Inference Dynamics

Latent Space Model for Higher-order Networks and Generalized Tensor Decomposition

Learning Topic Hierarchies by Tree-Directed Latent Variable Models

Infinite max-margin factor analysis via data augmentation

Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference

Advances on BYY Harmony Learning: Information Theoretic Perspective, Generalized Projection Geometry, and Independent Factor Autodetermination

Bayesian Ying Yang System, Best Harmony Learning, And Gaussian Manifold Based Family

NeuLFT: A Novel Approach to Nonlinear Canonical Polyadic Decomposition on High-Dimensional Incomplete Tensors

Learning Beyond Predefined Label Space Via Bayesian Nonparametric Topic Modelling

Diversity-Promoting Bayesian Learning of Latent Variable Models.