Abstract:This article considers Bayesian model selection via mean-field (MF) variational approximation. Towards this goal, we study the non-asymptotic properties of MF inference under the Bayesian framework that allows latent variables and model mis-specification. Concretely, we show a Bernstein von-Mises (BvM) theorem for the variational distribution from MF under possible model mis-specification, which implies the distributional convergence of MF variational approximation to a normal distribution centering at the maximal likelihood estimator (within the specified model). Motivated by the BvM theorem, we propose a model selection criterion using the evidence lower bound (ELBO), and demonstrate that the model selected by ELBO tends to asymptotically agree with the one selected by the commonly used Bayesian information criterion (BIC) as sample size tends to infinity. Comparing to BIC, ELBO tends to incur smaller approximation error to the log-marginal likelihood (a.k.a. model evidence) due to a better dimension dependence and full incorporation of the prior information. Moreover, we show the geometric convergence of the coordinate ascent variational inference (CAVI) algorithm under the parametric model framework, which provides a practical guidance on how many iterations one typically needs to run when approximating the ELBO. These findings demonstrate that variational inference is capable of providing a computationally efficient alternative to conventional approaches in tasks beyond obtaining point estimates, which is also empirically demonstrated by our extensive numerical experiments.

Robust Bayesian model selection for heavy-tailed linear regression using finite mixtures

A Novel Bayesian Robust Model and Its Application for Fault Detection and Automatic Supervision of Nonlinear Process

Bayesian estimation and prediction for certain mixtures

Model selection in finite mixture of regression models: a Bayesian approach with innovative weighted g priors and reversible jump Markov chain Monte Carlo implementation (vol 85, pg 2456, 2015)

Robust Bayesian nonparametric variable selection for linear regression

Flexible Bayesian quantile regression for nonlinear mixed effects models based on the generalized asymmetric Laplace distribution

Bayesian estimation for longitudinal data in a joint model with HPCs

A novel Bayesian method for variable selection and estimation in binary quantile regression

Bayesian variable selection in linear regression models with non-normal errors

Bayesian Model Selection in High-Dimensional Settings

Multimode process data modeling: A Dirichlet process mixture model based Bayesian robust factor analyzer approach

Bayesian analysis of linear regression models with autoregressive symmetrical errors and incomplete data

Simultaneous Bayesian Clustering and Model Selection with Mixture of Robust Factor Analyzers

Nonparametric Bayesian modeling on infinite mixture Student t copulas

Robust Bayesian Model Averaging for Linear Regression Models With Heavy-Tailed Errors

Model selection of Gaussian mixture process and its application

Bayesian Model Selection via Mean-Field Variational Approximation

Heavy-Tailed Pitman--Yor Mixture Models

t distribution-based robust semiparametric mixture regression model

Heavy-tailed Bayesian nonparametric adaptation

Model Selection for Exponential Power Mixture Regression Models