Abstract:Abstract Background The replication crisis hit the medical sciences about a decade ago, but today still most of the flaws inherent in null hypothesis significance testing (NHST) have not been solved. While the drawbacks of p -values have been detailed in endless venues, for clinical research, only a few attractive alternatives have been proposed to replace p -values and NHST. Bayesian methods are one of them, and they are gaining increasing attention in medical research, as some of their advantages include the description of model parameters in terms of probability, as well as the incorporation of prior information in contrast to the frequentist framework. While Bayesian methods are not the only remedy to the situation, there is an increasing agreement that they are an essential way to avoid common misconceptions and false interpretation of study results. The requirements necessary for applying Bayesian statistics have transitioned from detailed programming knowledge into simple point-and-click programs like JASP. Still, the multitude of Bayesian significance and effect measures which contrast the gold standard of significance in medical research, the p -value, causes a lack of agreement on which measure to report. Methods Therefore, in this paper, we conduct an extensive simulation study to compare common Bayesian significance and effect measures which can be obtained from a posterior distribution. In it, we analyse the behaviour of these measures for one of the most important statistical procedures in medical research and in particular clinical trials, the two-sample Student’s (and Welch’s) t-test. Results The results show that some measures cannot state evidence for both the null and the alternative. While the different indices behave similarly regarding increasing sample size and noise, the prior modelling influences the obtained results and extreme priors allow for cherry-picking similar to p-hacking in the frequentist paradigm. The indices behave quite differently regarding their ability to control the type I error rates and regarding their ability to detect an existing effect. Conclusion Based on the results, two of the commonly used indices can be recommended for more widespread use in clinical and biomedical research, as they improve the type I error control compared to the classic two-sample t-test and enjoy multiple other desirable properties.

Interval estimation, point estimation, and null hypothesis significance testing calibrated by an estimated posterior probability of the null hypothesis

Fiducialize statistical significance: transforming p-values into conservative posterior probabilities and Bayes factors

Discuss practical importance of results based on interval estimates and p-value functions, not only on point estimates and null p-values

Confidence distributions and hypothesis testing

Estimating the null distribution for conditional inference and genome-scale screening

A unified Bayesian framework for interval hypothesis testing in clinical trials

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

B-Value and Empirical Equivalence Bound: A New Procedure of Hypothesis Testing

P values, confidence intervals, or confidence levels for hypotheses?

Theoretical Justification of the Bi Error Method

The flaw of averages: Bayes factors as posterior means of the likelihood ratio

Post-hoc Hypothesis Testing

Empirical Bayes factors for common hypothesis tests

A Change-Point Approach to Estimating the Proportion of False Null Hypotheses in Multiple Testing

Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research

Multiple testing of interval composite null hypotheses using randomized p-values

Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Beyond 'statistical significance': A nontechnical primer of Bayesian statistics and Bayes factors for health researchers

Semantic and Cognitive Tools to Aid Statistical Science: Replace Confidence and Significance by Compatibility and Surprise

Statistical significance testing for mixed priors: a combined Bayesian and frequentist analysis

How sure are we? Two approaches to statistical inference