Abstract:In many hypothesis testing applications, we have mixed priors, with well-motivated informative priors for some parameters but not for others. The Bayesian methodology uses the Bayes factor and is helpful for the informative priors, as it incorporates Occam's razor via multiplicity or trials factor in the Look Elsewhere Effect. However, if the prior is not known completely, the frequentist hypothesis test via the false positive rate is a better approach, as it is less sensitive to the prior choice. We argue that when only partial prior information is available, it is best to combine the two methodologies by using the Bayes factor as a test statistic in the frequentist analysis. We show that the standard frequentist likelihood-ratio test statistic corresponds to the Bayes factor with a non-informative Jeffrey's prior. We also show that mixed priors increase the statistical power in frequentist analyses over the likelihood ratio test statistic. We develop an analytic formalism that does not require expensive simulations using a statistical mechanics approach to hypothesis testing in Bayesian and frequentist statistics. We introduce the counting of states in a continuous parameter space using the uncertainty volume as the quantum of the state. We show that both the p-value and Bayes factor can be expressed as energy versus entropy competition. We present analytic expressions that generalize Wilks' theorem beyond its usual regime of validity and work in a non-asymptotic regime. In specific limits, the formalism reproduces existing expressions, such as the p-value of linear models and periodograms. We apply the formalism to an example of exoplanet transits, where multiplicity can be more than $10^7$. We show that our analytic expressions reproduce the p-values derived from numerical simulations.

Detecting and diagnosing prior and likelihood sensitivity with power-scaling

Sensitivity analysis for Bayesian hierarchical models

Variance-based sensitivity of Bayesian inverse problems to the prior distribution

Sensitivity-Aware Amortized Bayesian Inference

Normalized Power Prior Bayesian Analysis

Bayesian sensitivity analysis for a missing data model

Statistical significance testing for mixed priors: a combined Bayesian and frequentist analysis

The Importance of Prior Sensitivity Analysis in Bayesian Statistics: Demonstrations Using an Interactive Shiny App

Prior sample size extensions for assessing prior impact and prior--likelihood discordance

Hyper-differential sensitivity analysis with respect to model discrepancy: Posterior Optimal Solution Sampling

BayesPPD: An R Package for Bayesian Sample Size Determination Using the Power and Normalized Power Prior for Generalized Linear Models

Multiscale model diagnostics

Bayes Factor in One-Sample Tests of Means with a Sensitivity Analysis: A Discussion of Separate Prior Distributions

Towards a power analysis for PLS-based methods

A likelihood-based sensitivity analysis for addressing publication bias in meta-analysis of diagnostic studies using exact likelihood

Improved marginal likelihood estimation via power posteriors and importance sampling

Weakly informative priors and prior-data conflict checking for likelihood-free inference

Bayesian design of clinical trials using the scale transformed power prior

A Bayesian inference and model selection algorithm with an optimisation scheme to infer the model noise power

Maximum Likelihood Estimation of Flexible Survival Densities with Importance Sampling

A multiple testing framework for diagnostic accuracy studies with co-primary endpoints