Abstract:In many hypothesis testing applications, we have mixed priors, with well-motivated informative priors for some parameters but not for others. The Bayesian methodology uses the Bayes factor and is helpful for the informative priors, as it incorporates Occam's razor via multiplicity or trials factor in the Look Elsewhere Effect. However, if the prior is not known completely, the frequentist hypothesis test via the false positive rate is a better approach, as it is less sensitive to the prior choice. We argue that when only partial prior information is available, it is best to combine the two methodologies by using the Bayes factor as a test statistic in the frequentist analysis. We show that the standard frequentist likelihood-ratio test statistic corresponds to the Bayes factor with a non-informative Jeffrey's prior. We also show that mixed priors increase the statistical power in frequentist analyses over the likelihood ratio test statistic. We develop an analytic formalism that does not require expensive simulations using a statistical mechanics approach to hypothesis testing in Bayesian and frequentist statistics. We introduce the counting of states in a continuous parameter space using the uncertainty volume as the quantum of the state. We show that both the p-value and Bayes factor can be expressed as energy versus entropy competition. We present analytic expressions that generalize Wilks' theorem beyond its usual regime of validity and work in a non-asymptotic regime. In specific limits, the formalism reproduces existing expressions, such as the p-value of linear models and periodograms. We apply the formalism to an example of exoplanet transits, where multiplicity can be more than $10^7$. We show that our analytic expressions reproduce the p-values derived from numerical simulations.

Bayes Factors Based on Test Statistics and Non-Local Moment Prior Densities

Bayesian Hypothesis Tests Using Nonparametric Statistics

On Bayes factors for hypothesis tests

Empirical Bayes factors for common hypothesis tests

Bayes factor functions for reporting outcomes of hypothesis tests

Bayes Factor in One-Sample Tests of Means with a Sensitivity Analysis: A Discussion of Separate Prior Distributions

Nonparametric Bayes Factors Based on Empirical Likelihood Ratios.

On the Null Distribution of Bayes Factors in Linear Regression

Testing High Dimensional Covariance Matrices Via Posterior Bayes Factor.

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

The flaw of averages: Bayes factors as posterior means of the likelihood ratio

A Two-Sample Test of High Dimensional Means Based on Posterior Bayes Factor

Testing Informative Hypotheses in Factor Analysis Models Using Bayes Factors.

Null hypothesis Bayes factor estimates can be biased in (some) common factorial designs: A simulation study

A Good check on the Bayes factor

An Objective and Robust Bayes Factor for the Hypothesis Test One Sample and Two Population Means

Statistical significance testing for mixed priors: a combined Bayesian and frequentist analysis

Using Bayes to get the most out of non-significant results

Distribution of Bayes' factor

What is a Bayes factor?

The Bayes Factor and Its Implementation in JASP: A Practical Primer