Approximating Bayes factors from minimal ANOVA summaries: An extension of the BIC method

Thomas J. Faulkenberry

DOI: https://doi.org/10.48550/arXiv.1710.02351

2017-10-06

Abstract:In this paper, I extend a method of Masson (2011) to develop an easy-to-use formula for performing Bayesian hypothesis tests from minimal ANOVA summaries.

Methodology

What problem does this paper attempt to address?

Estimating Bayes factors from minimal summary statistics in repeated measures analysis of variance designs

Thomas Faulkenberry

DOI: https://doi.org/10.51936/abic6583

2020-01-01

Advances in Methodology and Statistics

Abstract:In this paper, I develop a formula for estimating Bayes factors directly from minimal summary statistics produced in repeated measures analysis of variance designs. The formula, which requires knowing only the F-statistic, the number of subjects, and the number of repeated measurements per subject, is based on the BIC approximation of the Bayes factor, a common default method for Bayesian computation with linear models. In addition to providing computational examples, I report a simulation study in which I demonstrate that the formula compares favorably to a recently developed, more complex method that accounts for correlation between repeated measurements. The minimal BIC method provides a simple way for researchers to estimate Bayes factors from a minimal set of summary statistics, giving users a powerful index for estimating the evidential value of not only their own data, but also the data reported in published studies.

English Else
A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Xinyue Zhou,Joel R. Levin

DOI: https://doi.org/10.22237/jmasm/1099268100

2004-01-01

Journal of Modern Applied Statistical Methods

Abstract:This article extends Scheffé's modified (sequential) multiple-comparison procedure in one-way analysis-of-variance to other analysis situations, including interaction comparisons in factorial ANOVA designs, tests of partial regression coefficients in multiple-regression analysis, and comparisons of means in one-factor multivariate analyses of variance. Researchers who are concerned with maintaining familywise Type I error rates while increasing statistical power relative to the original (simultaneous) Scheffé-based procedures are encouraged to consider these improved multiple-comparison methods. Copyright © 2004 JMASM, Inc.
An automatic adaptive method to combine summary statistics in approximate Bayesian computation

Jonathan U Harrison,Ruth E Baker

DOI: https://doi.org/10.48550/arXiv.1703.02341

2018-08-02

Abstract:To infer the parameters of mechanistic models with intractable likelihoods, techniques such as approximate Bayesian computation (ABC) are increasingly being adopted. One of the main disadvantages of ABC in practical situations, however, is that parameter inference must generally rely on summary statistics of the data. This is particularly the case for problems involving high-dimensional data, such as biological imaging experiments. However, some summary statistics contain more information about parameters of interest than others, and it is not always clear how to weight their contributions within the ABC framework. We address this problem by developing an automatic, adaptive algorithm that chooses weights for each summary statistic. Our algorithm aims to maximize the distance between the prior and the approximate posterior by automatically adapting the weights within the ABC distance function. Computationally, we use a nearest neighbour estimator of the distance between distributions. We justify the algorithm theoretically based on properties of the nearest neighbour distance estimator. To demonstrate the effectiveness of our algorithm, we apply it to a variety of test problems, including several stochastic models of biochemical reaction networks, and a spatial model of diffusion, and compare our results with existing algorithms.

Computation
Bayesian mixtures of common factor analyzers: Model, variational inference, and applications

Xin Wei,Chunguang Li

DOI: https://doi.org/10.1016/j.sigpro.2013.04.007

IF: 4.729

2013-01-01

Signal Processing

Abstract:Recently, a representative approach, named mixtures of common factor analyzers (MCFA), was proposed for clustering high-dimensional observed data. Existing model-parameter estimation methods for this approach is based on the maximum likelihood criterion and performed by the expectation-maximization algorithm. In this paper, we consider the MCFA from a Bayesian perspective and propose the Bayesian mixtures of common factor analyzers (BMCFA) model, which replaces the deterministic model parameters in the MCFA by stochastic variables. Then we present a variational inference algorithm for this BMCFA model. Moreover, the proposed BMCFA model and the associated variational inference algorithm are used for clustering the high-dimensional synthetic data, the wine data from the UCI machine learning repository and the gene expression data. Experimental results illustrate that the BMCFA has good generalization capacities, automatically determining the appropriate number of clusters from high-dimensional observed data.
On Bayes factors for hypothesis tests

Karl Christoph Klauer,Constantin G. Meyer-Grant,David Kellen

DOI: https://doi.org/10.3758/s13423-024-02612-2

2024-11-28

Psychonomic Bulletin & Review

Abstract:We develop alternative families of Bayes factors for use in hypothesis tests as alternatives to the popular default Bayes factors. The alternative Bayes factors are derived for the statistical analyses most commonly used in psychological research – one-sample and two-sample t tests, regression, and ANOVA analyses. They possess the same desirable theoretical and practical properties as the default Bayes factors and satisfy additional theoretical desiderata while mitigating against two features of the default priors that we consider implausible. They can be conveniently computed via an R package that we provide. Furthermore, hypothesis tests based on Bayes factors and those based on significance tests are juxtaposed. This discussion leads to the insight that default Bayes factors as well as the alternative Bayes factors are equivalent to test-statistic-based Bayes factors as proposed by Johnson. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67 , 689–701. (2005). We highlight test-statistic-based Bayes factors as a general approach to Bayes-factor computation that is applicable to many hypothesis-testing problems for which an effect-size measure has been proposed and for which test power can be computed.

psychology, experimental, mathematical
A Bayes factor with reasonable model selection consistency for ANOVA model

Yuzo Maruyama

DOI: https://doi.org/10.48550/arXiv.0906.4329

2012-06-13

Abstract:For the ANOVA model, we propose a new g-prior based Bayes factor without integral representation, with reasonable model selection consistency for any asymptotic situations (either number of levels of the factor and/or number of replication in each level goes to infinity). Exact analytic calculation of the marginal density under a special choice of the priors enables such a Bayes factor.

Methodology
Bayes Factors for Comparison of Two-Way ANOVA Models

R. Vijayaragunathan,M. R. Srinivasan

DOI: https://doi.org/10.2991/jsta.d.201230.001

2021-01-01

Journal of Statistical Theory and Applications

Abstract:In the traditional two-way analysis of variance (ANOVA) model, it is possible to identify the significance of both the main effects and their interaction based on the P values. However, it is not possible to determine how much data supports the model when these effects are incorporated into the model. To overcome this practical difficulty, we applied...
Minimal sample size in balanced ANOVA models of crossed, nested, and mixed classifications

Bernhard Spangl,Norbert Kaiblinger,Peter Ruckdeschel,Dieter Rasch

DOI: https://doi.org/10.1080/03610926.2021.1938126

2021-06-22

Abstract:We consider balanced one-way, two-way, and three-way ANOVA models to test the hypothesis that the fixed factor A has no effect. The other factors are fixed or random. We determine the noncentrality parameter for the exact F-test, describe its minimal value by a sharp lower bound, and thus we can guarantee the worst-case power for the F-test. These results allow us to compute the minimal sample size, i.e. the minimal number of experiments needed. We also provide a structural result for the minimum sample size, proving a conjecture on the optimal experimental design.
A Good check on the Bayes factor

Nikola Sekulovski,Maarten Marsman,Eric-Jan Wagenmakers

DOI: https://doi.org/10.3758/s13428-024-02491-4

2024-09-04

Abstract:Bayes factor hypothesis testing provides a powerful framework for assessing the evidence in favor of competing hypotheses. To obtain Bayes factors, statisticians often require advanced, non-standard tools, making it important to confirm that the methodology is computationally sound. This paper seeks to validate Bayes factor calculations by applying two theorems attributed to Alan Turing and Jack Good. The procedure entails simulating data sets under two hypotheses, calculating Bayes factors, and assessing whether their expected values align with theoretical expectations. We illustrate this method with an ANOVA example and a network psychometrics application, demonstrating its efficacy in detecting calculation errors and confirming the computational correctness of the Bayes factor results. This structured validation approach aims to provide researchers with a tool to enhance the credibility of Bayes factor hypothesis testing, fostering more robust and trustworthy scientific inferences.
Empirical Bayes factors for common hypothesis tests

Frank Dudbridge

DOI: https://doi.org/10.1371/journal.pone.0297874

IF: 3.7

2024-02-23

PLoS ONE

Abstract:Bayes factors for composite hypotheses have difficulty in encoding vague prior knowledge, as improper priors cannot be used and objective priors may be subjectively unreasonable. To address these issues I revisit the posterior Bayes factor, in which the posterior distribution from the data at hand is re-used in the Bayes factor for the same data. I argue that this is biased when calibrated against proper Bayes factors, but propose adjustments to allow interpretation on the same scale. In the important case of a regular normal model, the bias in log scale is half the number of parameters. The resulting empirical Bayes factor is closely related to the widely applicable information criterion. I develop test-based empirical Bayes factors for several standard tests and propose an extension to multiple testing closely related to the optimal discovery procedure. When only a P -value is available, an approximate empirical Bayes factor is 10 p . I propose interpreting the strength of Bayes factors on a logarithmic scale with base 3.73, reflecting the sharpest distinction between weaker and stronger belief. This provides an objective framework for interpreting statistical evidence, and realises a Bayesian/frequentist compromise.

multidisciplinary sciences
Nonparametric Bayesian Negative Binomial Factor Analysis

Mingyuan Zhou

DOI: https://doi.org/10.48550/arXiv.1604.07464

2017-10-05

Abstract:A common approach to analyze a covariate-sample count matrix, an element of which represents how many times a covariate appears in a sample, is to factorize it under the Poisson likelihood. We show its limitation in capturing the tendency for a covariate present in a sample to both repeat itself and excite related ones. To address this limitation, we construct negative binomial factor analysis (NBFA) to factorize the matrix under the negative binomial likelihood, and relate it to a Dirichlet-multinomial distribution based mixed-membership model. To support countably infinite factors, we propose the hierarchical gamma-negative binomial process. By exploiting newly proved connections between discrete distributions, we construct two blocked and a collapsed Gibbs sampler that all adaptively truncate their number of factors, and demonstrate that the blocked Gibbs sampler developed under a compound Poisson representation converges fast and has low computational complexity. Example results show that NBFA has a distinct mechanism in adjusting its number of inferred factors according to the sample lengths, and provides clear advantages in parsimonious representation, predictive power, and computational complexity over previously proposed discrete latent variable models, which either completely ignore burstiness, or model only the burstiness of the covariates but not that of the factors.

Methodology,Machine Learning
An easy-to-use empirical likelihood ABC method

Sanjay Chaudhuri,Subhro Ghosh,David J. Nott,Kim Cuc Pham

DOI: https://doi.org/10.48550/arXiv.1810.01675

2018-10-08

Abstract:Many scientifically well-motivated statistical models in natural, engineering and environmental sciences are specified through a generative process, but in some cases it may not be possible to write down a likelihood for these models analytically. Approximate Bayesian computation (ABC) methods, which allow Bayesian inference in these situations, are typically computationally intensive. Recently, computationally attractive empirical likelihood based ABC methods have been suggested in the literature. These methods heavily rely on the availability of a set of suitable analytically tractable estimating equations. We propose an easy-to-use empirical likelihood ABC method, where the only inputs required are a choice of summary statistic, it's observed value, and the ability to simulate summary statistics for any parameter value under the model. It is shown that the posterior obtained using the proposed method is consistent, and its performance is explored using various examples.

Computation,Applications,Methodology,Machine Learning
Bayesian Hypothesis Tests Using Nonparametric Statistics

Ying Yuan,Valen E. Johnson

IF: 1.4

2008-01-01

Statistica Sinica

Abstract:Traditionally, the application of Bayesian testing procedures to classical nonparametric settings has been restricted by difficulties associated with prior specification, prohibitively expensive computation, and the absence of sampling densities for data. To overcome these difficulties, we model the sampling distributions of nonparametric test statistics-rather than the sampling distributions of original data-to obtain the Bayes factors required for Bayesian hypothesis tests. We apply this methodology to construct Bayes factors from a wide class of nonparametric test statistics having limiting normal distributions and illustrate these methods with data. Finally, we consider the extension of our methodology to nonparametric test statistics having limiting chi(2) distributions.
Model Selection for Mixtures of Factor Analyzers via Hierarchical BIC

Jianhua Zhao,Philip L. H. Yu,Lei Shi

2013-01-01

Abstract:Bayesian information criterion (BIC) is a common model selection criterion for mixtures of factor analyzers (MFA). However, it is found that BIC penalizes each factor analyzer implausibly using the whole sample size. In this paper, we propose a new criterion for MFA called hierarchical BIC (H-BIC). Formally, the main difference from BIC is that H-BIC penalizes each factor analyzer using its own effective sample size only. Theoretically, we show that HBIC is a large sample approximation of variational Bayesian (VB) lower bound and BIC is a further approximation of HBIC. Additionally, to perform H-BIC efficiently, we propose a novel algorithm in which we does not use H-BIC as a criterion to choose one from a set of candidate models with different latent dimensions, rather, we integrates the determination of latent dimensions into parameter estimation for a given number of components. Consequently, this algorithm only requires choosing one from the much smaller set with different number of components. Experiments on a number of synthetic and real data sets reveal that (i) H-BIC is more accurate than BIC and several existing competing methods; (ii) the proposed novel algorithm is much more efficient than that usually used for BIC.
Nonparametric Bayes Factors Based on Empirical Likelihood Ratios.

Albert Vexler,Wei Deng,Gregory E. Wilding

DOI: https://doi.org/10.1016/j.jspi.2012.08.011

IF: 1.095

2013-01-01

Journal of Statistical Planning and Inference

Abstract:Bayes methodology provides posterior distribution functions based on parametric likelihoods adjusted for prior distributions. A distribution-free alternative to the parametric likelihood is use of empirical likelihood (EL) techniques, well known in the context of nonparametric testing of statistical hypotheses. Empirical likelihoods have been shown to exhibit many of the properties of conventional parametric likelihoods. In this article, we propose and examine Bayes factors (BF) methods that are derived via the EL ratio approach. Following Kass & Wasserman [10], we consider Bayes factors type decision rules in the context of standard statistical testing techniques. We show that the asymptotic properties of the proposed procedure are similar to the classical BF's asymptotic operating characteristics. Although we focus on hypothesis testing, the proposed approach also yields confidence interval estimators of unknown parameters. Monte Carlo simulations were conducted to evaluate the theoretical results as well as to demonstrate the power of the proposed test.
A note on variational Bayesian factor analysis.

Jian-hua Zhao,Philip L H Yu

DOI: https://doi.org/10.1016/j.neunet.2008.11.002

IF: 7.8

2009-01-01

Neural Networks

Abstract:Existing works on variational bayesian (VB) treatment for factor analysis (FA) model such as [Ghahramani, Z., & Beal, M. (2000). Variational inference for Bayesian mixture of factor analysers. In Advances in neural information proceeding systems. Cambridge, MA: MIT Press; Nielsen, F. B. (2004). Variational approach to factor analysis and related models. Master's thesis, The Institute of Informatics and Mathematical Modelling, Technical University of Denmark.] are found theoretically and empirically to suffer two problems: (1) penalize the model more heavily than BIC and (2) perform unsatisfactorily in low noise cases as redundant factors can not be effectively suppressed. A novel VB treatment is proposed in this paper to resolve the two problems and a simulation study is conducted to testify its improved performance over existing treatments.
Approximate inferences for Bayesian hierarchical generalised linear regression models

Brandon Berman,Wesley O. Johnson,Weining Shen

DOI: https://doi.org/10.1111/anzs.12412

2024-05-09

Australian & New Zealand Journal of Statistics

Abstract:Summary Generalised linear mixed regression models are fundamental in statistics. Modelling random effects that are shared by individuals allows for correlation among those individuals. There are many methods and statistical packages available for analysing data using these models. Most require some form of numerical or analytic approximation because the likelihood function generally involves intractable integrals over the latents. The Bayesian approach avoids this issue by iteratively sampling the full conditional distributions for various blocks of parameters and latent random effects. Depending on the choice of the prior, some full conditionals are recognisable while others are not. In this paper we develop a novel normal approximation for the random effects full conditional, establish its asymptotic correctness and evaluate how well it performs. We make the case for hierarchical binomial and Poisson regression models with canonical link functions, for hierarchical gamma regression models with log link and for other cases. We also develop what we term a sufficient reduction (SR) approach to the Markov Chain Monte Carlo algorithm that allows for making inferences about all model parameters by replacing the full conditional for the latent variables with a considerably reduced dimensional function of the latents. We expect that this approximation could be quite useful in situations where there are a very large number of latent effects, which may be occurring in an increasingly 'Big Data' world. In the sequel, we compare our methods with INLA, which is a particularly popular method and which has been shown to be excellent in terms of speed and accuracy across a variety of settings. Our methods appear to be comparable to theirs in terms of accuracy, while INLA was faster, for the settings we considered. In addition, we note that our methods and those of others that involve Gibbs sampling trivially handle parameters that are functions of multiple parameters, while INLA approximations do not. Our primary illustration is for a three‐level hierarchical binomial regression model for data on health outcomes for patients who are clustered within physicians who are clustered within particular hospitals or hospital systems.

statistics & probability
On the importance of factorization for fast binned likelihood inference

César,Jesús-Valls

2024-01-22

Abstract:Likelihood-based inference, central in modern particle physics data analysis requires the extensive evaluation of a likelihood function that depends on set of parameters defined by the statistical model under consideration. If an analytical expression for the likelihood can be defined from first principles the procedure is computationally straightforward. However, most experiments require approximating the likelihood numerically using large statistical samples of synthetic events generated using Monte Carlo methods. As a result, the likelihood consists of a comparison of the expected versus the observed event rates in a collection of histogram bins, defining binned likelihood functions. When this occurs, evaluating the likelihood function involves, on each occasion, recalculating the prediction in those bins, increasing the computational load of these analysis drastically. In this text, I highlight the importance of identifying which are the unique event configurations in the binned likelihood definition and I provide an exact formula to update the event rate predictions utilizing the minimum number of necessary calculations by means of factorization. The aim of the discussion is to decrease the computational load of widespread high-energy physics analyses, leading to substantial speed improvements and reduced carbon footprints.

High Energy Physics - Experiment
Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian analysis of variance.

Daniel J. Schad,Bruno Nicenboim,Shravan Vasishth

DOI: https://doi.org/10.1037/met0000621

IF: 10.929

2024-01-25

Psychological Methods

Abstract:Bayesian linear mixed-effects models (LMMs) and Bayesian analysis of variance (ANOVA) are increasingly being used in the cognitive sciences to perform null hypothesis tests, where a null hypothesis that an effect is zero is compared with an alternative hypothesis that the effect exists and is different from zero. While software tools for Bayes factor null hypothesis tests are easily accessible, how to specify the data and the model correctly is often not clear. In Bayesian approaches, many authors use data aggregation at the by-subject level and estimate Bayes factors on aggregated data. Here, we use simulation-based calibration for model inference applied to several example experimental designs to demonstrate that, as with frequentist analysis, such null hypothesis tests on aggregated data can be problematic in Bayesian analysis. Specifically, when random slope variances differ (i.e., violated sphericity assumption), Bayes factors are too conservative for contrasts where the variance is small and they are too liberal for contrasts where the variance is large. Running Bayesian ANOVA on aggregated data can-if the sphericity assumption is violated-likewise lead to biased Bayes factor results. Moreover, Bayes factors for by-subject aggregated data are biased (too liberal) when random item slope variance is present but ignored in the analysis. These problems can be circumvented or reduced by running Bayesian LMMs on nonaggregated data such as on individual trials, and by explicitly modeling the full random effects structure. Reproducible code is available from https://osf.io/mjf47/. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

psychology, multidisciplinary
Bayes factor functions for reporting outcomes of hypothesis tests

Valen E. Johnson,Sandipan Pramanik,Rachael Shudde

DOI: https://doi.org/10.1073/pnas.2217331120

2022-10-01

Abstract:Bayes factors represent the ratio of probabilities assigned to data by competing scientific hypotheses. Drawbacks of Bayes factors are their dependence on prior specifications that define null and alternative hypotheses and difficulties encountered in their computation. To address these problems, we define Bayes factor functions (BFF) directly from common test statistics. BFFs depend on a single non-centrality parameter that can be expressed as a function of standardized effect sizes, and plots of BFFs versus effect size provide informative summaries of hypothesis tests that can be easily aggregated across studies. Such summaries eliminate the need for arbitrary P-value thresholds to define ``statistical significance.'' BFFs are available in closed form and can be computed easily from z, t, chi-squared, and F statistics.

Statistics Theory

Approximating Bayes factors from minimal ANOVA summaries: An extension of the BIC method

Estimating Bayes factors from minimal summary statistics in repeated measures analysis of variance designs

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

An automatic adaptive method to combine summary statistics in approximate Bayesian computation

Bayesian mixtures of common factor analyzers: Model, variational inference, and applications

On Bayes factors for hypothesis tests

A Bayes factor with reasonable model selection consistency for ANOVA model

Bayes Factors for Comparison of Two-Way ANOVA Models

Minimal sample size in balanced ANOVA models of crossed, nested, and mixed classifications

A Good check on the Bayes factor

Empirical Bayes factors for common hypothesis tests

Nonparametric Bayesian Negative Binomial Factor Analysis

An easy-to-use empirical likelihood ABC method

Bayesian Hypothesis Tests Using Nonparametric Statistics

Model Selection for Mixtures of Factor Analyzers via Hierarchical BIC

Nonparametric Bayes Factors Based on Empirical Likelihood Ratios.

A note on variational Bayesian factor analysis.

Approximate inferences for Bayesian hierarchical generalised linear regression models

On the importance of factorization for fast binned likelihood inference

Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian analysis of variance.

Bayes factor functions for reporting outcomes of hypothesis tests