Abstract:Existing Bayesian models, especially nonparametric Bayesian methods, rely on specially conceived priors to incorporate domain knowledge for discovering improved latent representations. While priors can affect posterior distributions through Bayes' rule, imposing posterior regularization is arguably more direct and in some cases more natural and general. In this paper, we present regularized Bayesian inference (RegBayes), a novel computational framework that performs posterior inference with a regularization term on the desired post-data posterior distribution under an information theoretical formulation. RegBayes is more flexible than the procedure that elicits expert knowledge via priors, and it covers both directed Bayesian networks and undirected Markov networks whose Bayesian formulation results in hybrid chain graph models. When the regularization is induced from a linear operator on the posterior distributions, such as the expectation operator, we present a general convex-analysis theorem to characterize the solution of RegBayes. Furthermore, we present two concrete examples of RegBayes, infinite latent support vector machines (iLSVM) and multi-task infinite latent support vector machines (MT-iLSVM), which explore the large-margin idea in combination with a nonparametric Bayesian model for discovering predictive latent features for classification and multi-task learning, respectively. We present efficient inference methods and report empirical studies on several benchmark datasets, which appear to demonstrate the merits inherited from both large-margin learning and Bayesian nonparametrics. Such results were not available until now, and contribute to push forward the interface between these two important subfields, which have been largely treated as isolated in the community.

Bayesian Probit Model with L-alpha and Elastic Net Regularization

Bayesian Regularized Quantile Regression

Flexible Bayesian quantile regression for nonlinear mixed effects models based on the generalized asymmetric Laplace distribution

Local Probabilistic Model for Bayesian Classification: a Generalized Local Classification Model

Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs

Variable selection and model prediction based on Lasso, adaptive lasso and elastic net

Bayesian Fused Lasso Modeling for Binary Data

Computationally Efficient Estimation of Large Probit Models

Bayesian Reward Models for LLM Alignment

Linearized Binary Regression

Bayesian Model Selection for High-Dimensional Ising Models, With Applications to Educational Data

Bayesian $L_{\frac{1}{2}}$ regression

A flexible Bayesian generalized linear model for dichotomous response data with an application to text categorization

Modelling Net Loan Loss with Bayesian and Frequentist Regression Analysis

Bayesian rank penalization.

Robust Bayesian Nonnegative Matrix Factorization with Implicit Regularizers

Robust Elastic Net Regression

Bayesian Model Selection in High-Dimensional Settings

Elastic Net Regularization Paths for All Generalized Linear Models

Large-Margin Regularized Softmax Cross-Entropy Loss

Robust Bayesian Model Averaging for Linear Regression Models With Heavy-Tailed Errors