Abstract:Under model misspecification, it is known that Bayesian posteriors often do not properly quantify uncertainty about true or pseudo-true parameters. Even more fundamentally, misspecification leads to a lack of reproducibility in the sense that the same model will yield contradictory posteriors on independent data sets from the true distribution. To define a criterion for reproducible uncertainty quantification under misspecification, we consider the probability that two confidence sets constructed from independent data sets have nonempty overlap, and we establish a lower bound on this overlap probability that holds for any valid confidence sets. We prove that credible sets from the standard posterior can strongly violate this bound, particularly in high-dimensional settings (i.e., with dimension increasing with sample size), indicating that it is not internally coherent under misspecification. To improve reproducibility in an easy-to-use and widely applicable way, we propose to apply bagging to the Bayesian posterior ("BayesBag"'); that is, to use the average of posterior distributions conditioned on bootstrapped datasets. We motivate BayesBag from first principles based on Jeffrey conditionalization and show that the bagged posterior typically satisfies the overlap lower bound. Further, we prove a Bernstein--Von Mises theorem for the bagged posterior, establishing its asymptotic normal distribution. We demonstrate the benefits of BayesBag via simulation experiments and an application to crime rate prediction.

Bayesian Posteriors Without Bayes' Theorem

The Bridged Posterior: Optimization, Profile Likelihood and a New Approach to Generalized Bayes

Bayesian inference: More than Bayes's theorem

Scalable Bayesian Learning with posteriors

Learning Bayesian Networks Using the Constrained Maximum a Posteriori Probability Method

Bayesian Joint Chance Constrained Optimization: Approximations and Statistical Consistency

Bayesian questions with frequentist answers

On permutation symmetries in Bayesian neural network posteriors: a variational perspective

Constrained inference through posterior projections

What Are Bayesian Neural Network Posteriors Really Like?

Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projections

PAC-Bayesian Theory Meets Bayesian Inference

A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and Asymptotics

On Cold Posteriors of Probabilistic Neural Networks: Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees

Mathematical theory of Bayesian statistics for unknown information source

Bayesian Quantile Estimation and Regression with Martingale Posteriors

The E-Posterior

A Channel-Based Perspective on Conjugate Priors

Bayes Hilbert Spaces for Posterior Approximation

Analytic Posteriors for Pearson's Correlation Coefficient

Reproducible Parameter Inference Using Bagged Posteriors