Abstract:This article considers Bayesian model selection via mean-field (MF) variational approximation. Towards this goal, we study the non-asymptotic properties of MF inference under the Bayesian framework that allows latent variables and model mis-specification. Concretely, we show a Bernstein von-Mises (BvM) theorem for the variational distribution from MF under possible model mis-specification, which implies the distributional convergence of MF variational approximation to a normal distribution centering at the maximal likelihood estimator (within the specified model). Motivated by the BvM theorem, we propose a model selection criterion using the evidence lower bound (ELBO), and demonstrate that the model selected by ELBO tends to asymptotically agree with the one selected by the commonly used Bayesian information criterion (BIC) as sample size tends to infinity. Comparing to BIC, ELBO tends to incur smaller approximation error to the log-marginal likelihood (a.k.a. model evidence) due to a better dimension dependence and full incorporation of the prior information. Moreover, we show the geometric convergence of the coordinate ascent variational inference (CAVI) algorithm under the parametric model framework, which provides a practical guidance on how many iterations one typically needs to run when approximating the ELBO. These findings demonstrate that variational inference is capable of providing a computationally efficient alternative to conventional approaches in tasks beyond obtaining point estimates, which is also empirically demonstrated by our extensive numerical experiments.

Minimum message length inference of the Poisson and geometric models using heavy-tailed prior distributions

Minimum Message Length Inference and Mixture Modelling of Inverse Gaussian Distributions

Minimum description length induction, Bayesianism, and Kolmogorov complexity

Multivariate‐bounded Gaussian mixture model with minimum message length criterion for model selection

Minimum Message Length Inference of the Exponential Distribution with Type I Censoring

Learning a Mixture Model for Clustering with the Completed Likelihood Minimum Message Length Criterion

MINIMUM DESCRIPTION LENGTH PRINCIPLE FOR LINEAR MIXED EFFECTS MODELS

Maximum Likelihood Inference in Robust Linear Mixed-Effects Models Using Multivariate T Distributions

Near Universal Consistency of the Maximum Pseudolikelihood Estimator for Discrete Models

Inference for compound truncated Poisson log-normal model with application to maximum precipitation data

Robust Bayesian model selection for heavy-tailed linear regression using finite mixtures

Mittag - Leffler function distribution - A new generalization of hyper-Poisson distribution

Statistical Inference for Generative Models with Maximum Mean Discrepancy

Minimaxity under half-Cauchy type priors

Mutual information maximization for amortized likelihood inference from sampled trajectories: MINIMALIST

Maximum Ideal Likelihood Estimator: An New Estimation and Inference Framework for Latent Variable Models

Advances in Bayesian model selection consistency for high-dimensional generalized linear models

Foundation of Calculating Normalized Maximum Likelihood for Continuous Probability Models

Modeling Sparse Data Using MLE with Applications to Microbiome Data

MLE convergence speed to information projection of exponential family: Criterion for model dimension and sample size -- complete proof version--

Bayesian Model Selection via Mean-Field Variational Approximation