Wrong Priors

Carlos C. Rodriguez

DOI: https://doi.org/10.1063/1.2821300

2007-09-07

Abstract:All priors are not created equal. There are right and there are wrong priors. That is the main conclusion of this contribution. I use, a cooked-up example designed to create drama, and a typical textbook example to show the pervasiveness of wrong priors in standard statistical practice.

Data Analysis, Statistics and Probability

What problem does this paper attempt to address?

The prior can generally only be understood in the context of the likelihood

Andrew Gelman,Daniel Simpson,Michael Betancourt

DOI: https://doi.org/10.3390/e19100555

2017-09-01

Abstract:A key sticking point of Bayesian analysis is the choice of prior distribution, and there is a vast literature on potential defaults including uniform priors, Jeffreys' priors, reference priors, maximum entropy priors, and weakly informative priors. These methods, however, often manifest a key conceptual tension in prior modeling: a model encoding true prior information should be chosen without reference to the model of the measurement process, but almost all common prior modeling techniques are implicitly motivated by a reference likelihood. In this paper we resolve this apparent paradox by placing the choice of prior into the context of the entire Bayesian analysis, from inference to prediction to model evaluation.

Methodology
Conditional probability and improper priors

Gunnar Taraldsen,Bo H. Lindqvist

DOI: https://doi.org/10.1080/03610926.2014.935430

2020-06-08

Abstract:The purpose of this paper is to present a mathematical theory that can be used as a foundation for statistics that include improper priors. This theory includes improper laws in the initial axioms and has in particular Bayes theorem as a consequence. Another consequence is that some of the usual calculation rules are modified. This is important in relation to common statistical practice which usually include improper priors, but tends to use unaltered calculation rules. In some cases the results are valid, but in other cases inconsistencies may appear. The famous marginalization paradoxes exemplify this latter case. An alternative mathematical theory for the foundations of statistics can be formulated in terms of conditional probability spaces. In this case the appearance of improper laws is a consequence of the theory. It is proved here that the resulting mathematical structures for the two theories are equivalent. The conclusion is that the choice of the first or the second formulation for the initial axioms can be considered a matter of personal preference. Readers that initially have concerns regarding improper priors can possibly be more open toward a formulation of the initial axioms in terms of conditional probabilities. The interpretation of an improper law is given by the corresponding conditional probabilities. Keywords: Axioms of statistics, Conditional probability space, Improper prior, Projective space

Statistics Theory
Properly constrained reference priors decay rates for efficient and robust posterior inference

Antoine Van Biesbroeck

2024-09-20

Abstract:In Bayesian analysis, reference priors are widely recognized for their objective nature. Yet, they often lead to intractable and improper priors, which complicates their application. Besides, informed prior elicitation methods are penalized by the subjectivity of the choices they require. In this paper, we aim at proposing a reconciliation of the aforementioned aspects. Leveraging the objective aspect of reference prior theory, we introduce two strategies of constraint incorporation to build tractable reference priors. One provides a simple and easy-to-compute solution when the improper aspect is not questioned, and the other introduces constraints to ensure the reference prior is proper, or it provides proper posterior. Our methodology emphasizes the central role of Jeffreys prior decay rates in this process, and the practical applicability of our results is demonstrated using an example taken from the literature.

Methodology,Statistics Theory
Overcoming priors anxiety

G. D'Agostini

DOI: https://doi.org/10.48550/arXiv.physics/9906048

1999-06-25

Abstract:The choice of priors may become an insoluble problem if priors and Bayes' rule are not seen and accepted in the framework of subjectivism. Therefore, the meaning and the role of subjectivity in science is considered and defended from the pragmatic point of view of an ``experienced scientist''. The case for the use of subjective priors is then supported and some recommendations for routine and frontier measurement applications are given. The issue of reference priors is also considered from the practical point of view and in the general context of ``Bayesian dogmatism''.

Data Analysis, Statistics and Probability,Probability
On a Class of Objective Priors from Scoring Rules

Fabrizio Leisen,Cristiano Villa,Stephen G. Walker

DOI: https://doi.org/10.48550/arXiv.1706.00599

2018-09-24

Abstract:Objective prior distributions represent an important tool that allows one to have the advantages of using the Bayesian framework even when information about the parameters of a model is not available. The usual objective approaches work off the chosen statistical model and in the majority of cases the resulting prior is improper, which can pose limitations to a practical implementation, even when the complexity of the model is moderate. In this paper we propose to take a novel look at the construction of objective prior distributions, where the connection with a chosen sampling distribution model is removed. We explore the notion of defining objective prior distributions which allow one to have some degree of flexibility, in particular in exhibiting some desirable features, such as being proper, or centered on specific values which would be of interest in nested model comparisons. The basic tool we use are proper scoring rules and the main result is a class of objective prior distributions that can be employed in scenarios where the usual model based priors fail, such as mixture models and model selection via Bayes factors. In addition, we show that the proposed class of priors is the result of minimising the information it contains, providing solid interpretation to the method.

Methodology,Applications,Other Statistics
Correcting Classifiers for Sample Selection Bias in Two-Phase Case-Control Studies

Norbert Krautenbacher,Fabian J. Theis,Christiane Fuchs

DOI: https://doi.org/10.1155/2017/7847531

IF: 2.809

2017-01-01

Computational and Mathematical Methods in Medicine

Abstract:Epidemiological studies often utilize stratified data in which rare outcomes or exposures are artificially enriched. This design can increase precision in association tests but distorts predictions when applying classifiers on nonstratified data. Several methods correct for this so-called sample selection bias, but their performance remains unclear especially for machine learning classifiers. With an emphasis on two-phase case-control studies, we aim to assess which corrections to perform in which setting and to obtain methods suitable for machine learning techniques, especially the randomforest. We propose two newresampling-basedmethods to resemble the original data and covariance structure: stochastic inverse-probability oversampling and parametric inverse-probability bagging. We compare all techniques for the random forest and other classifiers, both theoretically and on simulated and real data. Empirical results show that the random forest profits from only the parametric inverse-probability bagging proposed by us. For other classifiers, correction is mostly advantageous, and methods perform uniformly. We discuss consequences of inappropriate distribution assumptions and reason for different behaviors between the random forest and other classifiers. In conclusion, we provide guidance for choosing correction methods when training classifiers on biased samples. For random forests, our method outperforms state-of-the-art procedures if distribution assumptions are roughly fulfilled. We provide our implementation in the R package sambia.
Dangers of the Defaults: A Tutorial on the Impact of Default Priors When Using Bayesian SEM With Small Samples

Sanne C. Smid,Sonja D. Winter

DOI: https://doi.org/10.3389/fpsyg.2020.611963

IF: 3.8

2020-12-11

Frontiers in Psychology

Abstract:When Bayesian estimation is used to analyze Structural Equation Models (SEMs), prior distributions need to be specified for all parameters in the model. Many popular software programs offer default prior distributions, which is helpful for novel users and makes Bayesian SEM accessible for a broad audience. However, when the sample size is small, those prior distributions are not always suitable and can lead to untrustworthy results. In this tutorial, we provide a non-technical discussion of the risks associated with the use of default priors in small sample contexts. We discuss how default priors can unintentionally behave as highly informative priors when samples are small. Also, we demonstrate an online educational Shiny app, in which users can explore the impact of varying prior distributions and sample sizes on model results. We discuss how the Shiny app can be used in teaching; provide a reading list with literature on how to specify suitable prior distributions; and discuss guidelines on how to recognize (mis)behaving priors. It is our hope that this tutorial helps to spread awareness of the importance of specifying suitable priors when Bayesian SEM is used with small samples.

psychology, multidisciplinary
Robust priors for regularized regression

Sebastian Bobadilla-Suarez,Matt Jones,Bradley C. Love

DOI: https://doi.org/10.48550/arXiv.2010.02610

2021-10-25

Abstract:Induction benefits from useful priors. Penalized regression approaches, like ridge regression, shrink weights toward zero but zero association is usually not a sensible prior. Inspired by simple and robust decision heuristics humans use, we constructed non-zero priors for penalized regression models that provide robust and interpretable solutions across several tasks. Our approach enables estimates from a constrained model to serve as a prior for a more general model, yielding a principled way to interpolate between models of differing complexity. We successfully applied this approach to a number of decision and classification problems, as well as analyzing simulated brain imaging data. Models with robust priors had excellent worst-case performance. Solutions followed from the form of the heuristic that was used to derive the prior. These new algorithms can serve applications in data analysis and machine learning, as well as help in understanding how people transition from novice to expert performance.

Machine Learning
The posterior probability of a null hypothesis given a statistically significant result

Daniel J. Schad,Shravan Vasishth

DOI: https://doi.org/10.48550/arXiv.1901.06889

2022-04-16

Abstract:When researchers carry out a null hypothesis significance test, it is tempting to assume that a statistically significant result lowers Prob(H0), the probability of the null hypothesis being true. Technically, such a statement is meaningless for various reasons: e.g., the null hypothesis does not have a probability associated with it. However, it is possible to relax certain assumptions to compute the posterior probability Prob(H0) under repeated sampling. We show in a step-by-step guide that the intuitively appealing belief, that Prob(H0) is low when significant results have been obtained under repeated sampling, is in general incorrect and depends greatly on: (a) the prior probability of the null being true; (b) type-I error rate, (c) type-II error rate, and (d) replication of a result. Through step-by-step simulations using open-source code in the R System of Statistical Computing, we show that uncertainty about the null hypothesis being true often remains high despite a significant result. To help the reader develop intuitions about this common misconception, we provide a Shiny app (<a class="link-external link-https" href="https://danielschad.shinyapps.io/probnull/" rel="external noopener nofollow">this https URL</a>). We expect that this tutorial will help researchers better understand and judge results from null hypothesis significance tests.

Applications
A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Fabian Dablander,Karoline Huth,Quentin F. Gronau,Alexander Etz,Eric‐Jan Wagenmakers

DOI: https://doi.org/10.1002/sim.9278

2021-12-12

Statistics in Medicine

Abstract:Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains, it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis testing by means of the Bayes factor provides one avenue to do so, requiring the specification of prior distributions for parameters. The most popular analysis approach views the comparison of proportions from a contingency table perspective, assigning prior distributions directly to the two proportions. Another, less popular approach views the problem from a logistic regression perspective, assigning prior distributions to logit-transformed parameters. Reanalyzing 39 null results from the New England Journal of Medicine with both approaches, we find that they can lead to markedly different conclusions, especially when the observed proportions are at the extremes (ie, very low or very high). We explain these stark differences and provide recommendations for researchers interested in testing the equality of two proportions and users of Bayes factors more generally. The test that assigns prior distributions to logit-transformed parameters creates prior dependence between the two proportions and yields weaker evidence when the observations are at the extremes. When comparing two proportions, we argue that this test should become the new default.

public, environmental & occupational health,medicine, research & experimental,medical informatics,mathematical & computational biology,statistics & probability
Suspected serial killers and unsuspected statistical blunders

John O'Quigley,John O’Quigley

DOI: https://doi.org/10.1177/00258024241242549

2024-04-03

Medicine Science and the Law

Abstract:Medicine, Science and the Law, Ahead of Print. A whole branch of theoretical statistics devotes itself to the analysis of clusters, the aim being to distinguish an apparent cluster arising randomly from one that is more likely to have been produced as a result of some systematic influence. There are many examples in medicine and some that involve both medicine and the legal field; criminal law in particular. Observed clusters or a series of cases in a given setting can set off alarm bells, the recent conviction of Lucy Letby in England being an example. It was an observed cluster, a series of deaths among neonates, that prompted the investigation of Letby. There have been other similar cases in the past and there will be similar cases in the future. Our purpose is not to reconsider any particular trial but, rather, to work with similar, indeed more extreme numbers of cases as a way to underline the statistical mistakes that can be made when attempting to make sense of the data. These notions are illustrated via a made-up case of 10 incidents where the anticipated count was only 2. The most common statistical analysis would associate a probability of less than 0.00005 with this outcome: A very rare event. However, a more careful analysis that avoids common pitfalls results in a probability close to 0.5, indicating that, given the circumstances, we were as likely to see 10 or more as we were to see less than 10.

law,medicine, legal
Intrinsic tests for the equality of two correlated proportions

Guido Consonni,Luca La Rocca

DOI: https://doi.org/10.48550/arXiv.0707.3877

2007-07-26

Abstract:Correlated proportions arise in longitudinal (panel) studies. A typical example is the ``opinion swing'' problem: ``Has the proportion of people favoring a politician changed after his recent speech to the nation on TV?''. Since the same group of individuals is interviewed before and after the speech, the two proportions are correlated. A natural null hypothesis to be tested is whether the corresponding population proportions are equal. A standard Bayesian approach to this problem has already been considered in the literature, based on a Dirichlet prior for the cell-probabilities of the underlying two-by-two table under the alternative hypothesis, together with an induced prior under the null. In lack of specific prior information, a diffuse (e.g. uniform) distribution may be used. We claim that this approach is not satisfactory, since in a testing problem one should make sure that the prior under the alternative be adequately centered around the region specified by the null, in order to obtain a fair comparison between the two hypotheses. Following an intrinsic prior methodology, we develop two strategies for the construction of a collection of objective priors increasingly peaked around the null. We provide a simple interpretation of their structure in terms of weighted imaginary sample scenarios. We illustrate our method by means of three examples, carrying out sensitivity analysis and providing comparison with existing results.

Statistics Theory
Embracing the Paradigm Shift from Variable-Based to Case-Based Modeling

Arch G. Woodside

DOI: https://doi.org/10.1108/S1069-096420180000025003

2019-10-07

Advances in Business Marketing and Purchasing

Abstract:Currently, most of the empirical management, marketing, and psychology articles in the leading journals in these disciplines are examples of bad science practice. Bad science practice includes mismatching case (actor) focused theory and variable-data analysis with null hypothesis significance tests (NHST) of directional predictions (i.e., symmetric models proposing increases in each of several independent X 's associates with increases in a dependent Y ). Good science includes matching case-focused theory with case-focused data analytic tools and using somewhat precise outcome tests (SPOT) of asymmetric models. Good science practice achieves requisite variety necessary for deep explanation, description, and accurate prediction. Based on a thorough review of relevant literature, Hubbard (2016) concludes that reporting NHST results (e.g., an observed standardized partial regression betas for X 's differ from zero or that two means differ from zero) are examples of corrupt research. Hubbard (2017) expresses disappointment over the tepid response to his book. The pervasive teaching and use of NHST is one ingredient explaining the indifference, "I can't change just because it's [NHST] wrong." The fear of submission rejection is another reason for rejecting asymmetric modeling and SPOT. Reporting findings from both bad and good science practices may be necessary until asymmetric modeling and SPOT receive wider acceptance than held presently.
Sculpting priors

James Theiler

2024-08-09

Abstract:Bayesian priors are investigated for detecting targets of known spectral signature (but unknown strength) in cluttered backgrounds. A specific problem is the construction (or ``sculpting'') of a Bayesian prior that uniformly outperforms its non-Bayesian counterpart, the nominally sub-optimal but widely used Generalized Likelihood Ratio Test (GLRT).

Signal Processing,Statistics Theory
Connecting Simple and Precise P-values to Complex and Ambiguous Realities

Sander Greenland

DOI: https://doi.org/10.1111/sjos.12645

2023-09-13

Abstract:Mathematics is a limited component of solutions to real-world problems, as it expresses only what is expected to be true if all our assumptions are correct, including implicit assumptions that are omnipresent and often incorrect. Statistical methods are rife with implicit assumptions whose violation can be life-threatening when results from them are used to set policy. Among them are that there is human equipoise or unbiasedness in data generation, management, analysis, and reporting. These assumptions correspond to levels of cooperation, competence, neutrality, and integrity that are absent more often than we would like to believe. Given this harsh reality, we should ask what meaning, if any, we can assign to the P-values, 'statistical significance' declarations, 'confidence' intervals, and posterior probabilities that are used to decide what and how to present (or spin) discussions of analyzed data. By themselves, P-values and CI do not test any hypothesis, nor do they measure the significance of results or the confidence we should have in them. The sense otherwise is an ongoing cultural error perpetuated by large segments of the statistical and research community via misleading terminology. So-called 'inferential' statistics can only become contextually interpretable when derived explicitly from causal stories about the real data generator (such as randomization), and can only become reliable when those stories are based on valid and public documentation of the physical mechanisms that generated the data. Absent these assurances, traditional interpretations of statistical results become pernicious fictions that need to be replaced by far more circumspect descriptions of data and model relations.

Methodology,Quantitative Methods
Bayesian hypothesis tests with diffuse priors: Can we have our cake and eat it too?

J. T. Ormerod,M. Stewart,W. Yu,S. E. Romanes

DOI: https://doi.org/10.1111/anzs.12410

2024-03-20

Australian & New Zealand Journal of Statistics

Abstract:Summary We propose a new class of priors for Bayesian hypothesis testing, which we name 'cake priors'. These priors circumvent the Jeffreys–Lindley paradox (also called Bartlett's paradox) a problem associated with the use of diffuse priors leading to nonsensical statistical inferences. Cake priors allow the use of diffuse priors (having one's cake) while achieving theoretically justified inferences (eating it too). We demonstrate this methodology for Bayesian hypotheses tests for various common scenarios. The resulting Bayesian test statistic takes the form of a penalised likelihood ratio test statistic. Under typical regularity conditions, we show that Bayesian hypothesis tests based on cake priors are Chernoff consistent, that is, achieve zero type I and II error probabilities asymptotically. We also discuss Lindley's paradox and argue that the paradox occurs with small and vanishing probability as sample size increases.

statistics & probability
Bayesian reasoning and the prior in court: not legally normative but unavoidable

Ronald Meester,Lonneke Stevens

DOI: https://doi.org/10.1093/lpr/mgae001

2024-01-01

Law, Probability and Risk

Abstract:Abstract We introduce Bayesian reasoning in court not as a toolbox for doing computations, but as a way to assess evidence in a case. We argue that Bayesian reasoning comes naturally, even when the findings in a case cannot readily be translated into numbers. Not having numbers at one’s disposal is not an obstacle to use Bayesian reasoning. Although we present a coherent and complete view, we focus on the prior, since that seems to be the most problematic part of Bayesian reasoning. We explain that attempts to numerically express the prior fail in general, but also that a prior is necessary and cannot be dispensed with. Indeed, we explain in detail why decision-making should not be based on likelihood ratios alone. We next discuss two of the most delicate questions around the prior: (1) the possible conflict with the presumption of innocence, and (2) the idea that unwanted personal conviction (like racism) might enter the decision procedure via the prior. We conclude that these alleged problems are not problematic after all, and we carefully explain this position.

mathematics,statistics & probability,social sciences, mathematical methods,law
The experiment is just as important as the likelihood in understanding the prior: A cautionary note on robust cognitive modelling

Lauren Kennedy,Daniel Simpson,Andrew Gelman

DOI: https://doi.org/10.48550/arXiv.1905.10341

2019-05-24

Applications

Abstract:Cognitive modelling shares many features with statistical modelling, making it seem trivial to borrow from the practices of robust Bayesian statistics to protect the practice of robust cognitive modelling. We take one aspect of statistical workflow-prior predictive checks-and explore how they might be applied to a cognitive modelling task. We find that it is not only the likelihood that is needed to interpret the priors, we also need to incorporate experiment information as well. This suggests that while cognitive modelling might borrow from statistical practices, especially workflow, care must be made to make the adaptions necessary.
Elicitation, measuring bias, checking for prior-data conflict and inference with a Dirichlet prior

Michael Evans,Irwin Guttman,Peiying Li

DOI: https://doi.org/10.48550/arXiv.1703.03023

2017-03-09

Abstract:Methods are developed for eliciting a Dirichlet prior based upon bounds on the individual probabilities that hold with virtual certainty. This approach to selecting a prior is applied to a contingency table problem where it is demonstrated how to assess the bias in the prior as well as how to check for prior-data conflict. It is shown that the assessment of a hypothesis via relative belief can easily take into account what it means for the falsity of the hypothesis to correspond to a difference of practical importance and provide evidence in favor of a hypothesis.

Methodology
The Experiment is just as Important as the Likelihood in Understanding the Prior: a Cautionary Note on Robust Cognitive Modeling

Lauren Kennedy,Daniel Simpson,Andrew Gelman

DOI: https://doi.org/10.1007/s42113-019-00051-0

2019-10-07

Abstract:Cognitive modeling shares many features with statistical modeling, making it seem trivial to borrow from the practices of robust Bayesian statistics to protect the practice of robust cognitive modeling. We take one aspect of statistical workflow—prior predictive checks—and explore how they might be applied to a cognitive modeling task. We find that it is not only the likelihood that is needed to interpret the priors, we also need to incorporate experiment information as well. This suggests that while cognitive modeling might borrow from statistical practices, especially workflow, care must be taken to make the necessary adaptions.

Wrong Priors

The prior can generally only be understood in the context of the likelihood

Conditional probability and improper priors

Properly constrained reference priors decay rates for efficient and robust posterior inference

Overcoming priors anxiety

On a Class of Objective Priors from Scoring Rules

Correcting Classifiers for Sample Selection Bias in Two-Phase Case-Control Studies

Dangers of the Defaults: A Tutorial on the Impact of Default Priors When Using Bayesian SEM With Small Samples

Robust priors for regularized regression

The posterior probability of a null hypothesis given a statistically significant result

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Suspected serial killers and unsuspected statistical blunders

Intrinsic tests for the equality of two correlated proportions

Embracing the Paradigm Shift from Variable-Based to Case-Based Modeling

Sculpting priors

Connecting Simple and Precise P-values to Complex and Ambiguous Realities

Bayesian hypothesis tests with diffuse priors: Can we have our cake and eat it too?

Bayesian reasoning and the prior in court: not legally normative but unavoidable

The experiment is just as important as the likelihood in understanding the prior: A cautionary note on robust cognitive modelling

Elicitation, measuring bias, checking for prior-data conflict and inference with a Dirichlet prior

The Experiment is just as Important as the Likelihood in Understanding the Prior: a Cautionary Note on Robust Cognitive Modeling