Abstract:Existing Bayesian models, especially nonparametric Bayesian methods, rely on specially conceived priors to incorporate domain knowledge for discovering improved latent representations. While priors can affect posterior distributions through Bayes' rule, imposing posterior regularization is arguably more direct and in some cases more natural and general. In this paper, we present regularized Bayesian inference (RegBayes), a novel computational framework that performs posterior inference with a regularization term on the desired post-data posterior distribution under an information theoretical formulation. RegBayes is more flexible than the procedure that elicits expert knowledge via priors, and it covers both directed Bayesian networks and undirected Markov networks whose Bayesian formulation results in hybrid chain graph models. When the regularization is induced from a linear operator on the posterior distributions, such as the expectation operator, we present a general convex-analysis theorem to characterize the solution of RegBayes. Furthermore, we present two concrete examples of RegBayes, infinite latent support vector machines (iLSVM) and multi-task infinite latent support vector machines (MT-iLSVM), which explore the large-margin idea in combination with a nonparametric Bayesian model for discovering predictive latent features for classification and multi-task learning, respectively. We present efficient inference methods and report empirical studies on several benchmark datasets, which appear to demonstrate the merits inherited from both large-margin learning and Bayesian nonparametrics. Such results were not available until now, and contribute to push forward the interface between these two important subfields, which have been largely treated as isolated in the community.

Posterior Consistency of the Silverman G-Prior in Bayesian Model Choice.

Advances in Bayesian model selection consistency for high-dimensional generalized linear models

On Posterior Consistency of Tail Index for Bayesian Kernel Mixture Models

EP-GIG Priors and Applications in Bayesian Sparse Learning

Posterior consistency of Gaussian process prior for nonparametric binary regression

Bayesian Generalized Kernel Mixed Models

Bayesian estimation for longitudinal data in a joint model with HPCs

Posterior consistency of semi-supervised regression on graphs

Sparse reconstructions from few noisy data: analysis of hierarchical Bayesian models with generalized gamma hyperpriors

Compatibility of Prior Specifications Across Linear Models

Probabilistic Classifiers with a Generalized Gaussian Scale Mixture Prior

Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs

Generalized sparsity-promoting solvers for Bayesian inverse problems: Versatile sparsifying transforms and unknown noise variances

The prior can generally only be understood in the context of the likelihood

Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Model

Bernstein von-Mises Theorem for g-prior and nonlocal prior

Bayesian Model Selection in High-Dimensional Settings

Matching prior pairs connecting Maximum A Posteriori estimation and posterior expectation

Sparsity promoting hybrid solvers for hierarchical Bayesian inverse problems

The Bridged Posterior: Optimization, Profile Likelihood and a New Approach to Generalized Bayes

Bayesian graph selection consistency under model misspecification