Abstract:Abstract Background The replication crisis hit the medical sciences about a decade ago, but today still most of the flaws inherent in null hypothesis significance testing (NHST) have not been solved. While the drawbacks of p -values have been detailed in endless venues, for clinical research, only a few attractive alternatives have been proposed to replace p -values and NHST. Bayesian methods are one of them, and they are gaining increasing attention in medical research, as some of their advantages include the description of model parameters in terms of probability, as well as the incorporation of prior information in contrast to the frequentist framework. While Bayesian methods are not the only remedy to the situation, there is an increasing agreement that they are an essential way to avoid common misconceptions and false interpretation of study results. The requirements necessary for applying Bayesian statistics have transitioned from detailed programming knowledge into simple point-and-click programs like JASP. Still, the multitude of Bayesian significance and effect measures which contrast the gold standard of significance in medical research, the p -value, causes a lack of agreement on which measure to report. Methods Therefore, in this paper, we conduct an extensive simulation study to compare common Bayesian significance and effect measures which can be obtained from a posterior distribution. In it, we analyse the behaviour of these measures for one of the most important statistical procedures in medical research and in particular clinical trials, the two-sample Student’s (and Welch’s) t-test. Results The results show that some measures cannot state evidence for both the null and the alternative. While the different indices behave similarly regarding increasing sample size and noise, the prior modelling influences the obtained results and extreme priors allow for cherry-picking similar to p-hacking in the frequentist paradigm. The indices behave quite differently regarding their ability to control the type I error rates and regarding their ability to detect an existing effect. Conclusion Based on the results, two of the commonly used indices can be recommended for more widespread use in clinical and biomedical research, as they improve the type I error control compared to the classic two-sample t-test and enjoy multiple other desirable properties.

Bayesian Data Analysis: A Fresh Approach to Power Issues and Null Hypothesis Interpretation

Bayesian data analysis.

Beyond 'statistical significance': A nontechnical primer of Bayesian statistics and Bayes factors for health researchers

Bayesian Hypothesis Tests Using Nonparametric Statistics

An introduction to Bayesian statistics in health psychology

Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications

Why bother with Bayesian t-tests?

Bayesian alternatives to null hypothesis significance testing in biomedical research: a non-technical introduction to Bayesian inference with JASP

Bayesian statistics for clinical research

There is Nothing Magical about Bayesian Statistics: An Introduction to Epistemic Probabilities in Data Analysis for Psychology Starters

Practical Bayesian Inference in Neuroscience: Or How I Learned To Stop Worrying and Embrace the Distribution

Modern Bayesian Statistics in Clinical Research

Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence

Bayesian inference for psychology. Part II: Example applications with JASP

The Bayes Factor and Its Implementation in JASP: A Practical Primer

Comparing researchers' degree of dichotomous thinking using frequentist versus Bayesian null hypothesis testing

A frequentist two-sample test based on Bayesian model selection

Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research

Why do we need to employ Bayesian statistics and how can we employ it in studies of moral education?: With practical guidelines to use JASP for educators and researchers

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Bayesian tests of two proportions: A tutorial with R and JASP