Abstract:Existing Bayesian models, especially nonparametric Bayesian methods, rely on specially conceived priors to incorporate domain knowledge for discovering improved latent representations. While priors can affect posterior distributions through Bayes' rule, imposing posterior regularization is arguably more direct and in some cases more natural and general. In this paper, we present regularized Bayesian inference (RegBayes), a novel computational framework that performs posterior inference with a regularization term on the desired post-data posterior distribution under an information theoretical formulation. RegBayes is more flexible than the procedure that elicits expert knowledge via priors, and it covers both directed Bayesian networks and undirected Markov networks whose Bayesian formulation results in hybrid chain graph models. When the regularization is induced from a linear operator on the posterior distributions, such as the expectation operator, we present a general convex-analysis theorem to characterize the solution of RegBayes. Furthermore, we present two concrete examples of RegBayes, infinite latent support vector machines (iLSVM) and multi-task infinite latent support vector machines (MT-iLSVM), which explore the large-margin idea in combination with a nonparametric Bayesian model for discovering predictive latent features for classification and multi-task learning, respectively. We present efficient inference methods and report empirical studies on several benchmark datasets, which appear to demonstrate the merits inherited from both large-margin learning and Bayesian nonparametrics. Such results were not available until now, and contribute to push forward the interface between these two important subfields, which have been largely treated as isolated in the community.

Nonparametric Bayesian Multi-task Learning with Max-margin Posterior Regularization

Bayesian Max-margin Multi-Task Learning with Data Augmentation.

Nonparametric Bayesian Multi-Task Large-margin Classification.

Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs

Max-margin Based Bayesian Classifier.

Nonparametric Bayesian feature selection for multi-task learning

Learning Bayesian Network Parameters from Small Data Set: A Spatially Maximum a Posteriori Method.

Learning Bayesian Networks Using the Constrained Maximum a Posteriori Probability Method

Learning Bayesian Network Parameters Via Minimax Algorithm

Nonparametric Max-Margin Matrix Factorization for Collaborative Prediction.

Max-Margin Nonparametric Latent Feature Models for Link Prediction.

Bayesian Active Learning for Discrete Latent Variable Models

Infinite Latent SVM for Classification and Multi-task Learning.

Online Bayesian Max-Margin Subspace Learning for Multi-View Classification and Regression

Bayesian Maximum Margin Principal Component Analysis

Label-Sensitive Task Grouping by Bayesian Nonparametric Approach for Multi-Task Multi-Label Learning.

Online Bayesian Max-Margin Subspace Multi-View Learning.

Cross-Domain Multitask Learning with Latent Probit Models

Multi-view Learning as a Nonparametric Nonlinear Inter-Battery Factor Analysis

Nonparametric Bayesian Mixed-effect Model: a Sparse Gaussian Process Approach

Infinite max-margin factor analysis via data augmentation