Abstract:Abstract Background The replication crisis hit the medical sciences about a decade ago, but today still most of the flaws inherent in null hypothesis significance testing (NHST) have not been solved. While the drawbacks of p -values have been detailed in endless venues, for clinical research, only a few attractive alternatives have been proposed to replace p -values and NHST. Bayesian methods are one of them, and they are gaining increasing attention in medical research, as some of their advantages include the description of model parameters in terms of probability, as well as the incorporation of prior information in contrast to the frequentist framework. While Bayesian methods are not the only remedy to the situation, there is an increasing agreement that they are an essential way to avoid common misconceptions and false interpretation of study results. The requirements necessary for applying Bayesian statistics have transitioned from detailed programming knowledge into simple point-and-click programs like JASP. Still, the multitude of Bayesian significance and effect measures which contrast the gold standard of significance in medical research, the p -value, causes a lack of agreement on which measure to report. Methods Therefore, in this paper, we conduct an extensive simulation study to compare common Bayesian significance and effect measures which can be obtained from a posterior distribution. In it, we analyse the behaviour of these measures for one of the most important statistical procedures in medical research and in particular clinical trials, the two-sample Student’s (and Welch’s) t-test. Results The results show that some measures cannot state evidence for both the null and the alternative. While the different indices behave similarly regarding increasing sample size and noise, the prior modelling influences the obtained results and extreme priors allow for cherry-picking similar to p-hacking in the frequentist paradigm. The indices behave quite differently regarding their ability to control the type I error rates and regarding their ability to detect an existing effect. Conclusion Based on the results, two of the commonly used indices can be recommended for more widespread use in clinical and biomedical research, as they improve the type I error control compared to the classic two-sample t-test and enjoy multiple other desirable properties.

A comparison of Bayesian and score methods for interval estimates of positive/negative likelihood ratios in support of diagnostic device performance evaluation

A Robust Phenotype-Driven Likelihood Ratio Analysis Approach Assisting Interpretable Clinical Diagnosis of Rare Diseases.

Common odds ratio test and interval estimation for stratified bilateral and unilateral data

Bayesian Approach for Design and Analysis of Medical Device Trials in the Era of Modern Clinical Studies

Analysis of two Binomial Proportions in Non-inferiority Confirmatory Trials

A Bayesian approach for evaluating equivalence over multiple groups, and comparison with frequentist tost

Statistical inference for diagnostic test accuracy studies with multiple comparisons

The flaw of averages: Bayes factors as posterior means of the likelihood ratio

Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research

Interval estimation of common risk difference for stratified unilateral and bilateral data✩

Bayesian Solutions for Assessing Differential Effects in Biomarker Positive and Negative Subgroups

Bayesian bivariate meta-analysis of diagnostic test studies with interpretable priors

Homogeneity tests and interval estimations of risk differences for stratified bilateral and unilateral correlated data

A unified Bayesian framework for interval hypothesis testing in clinical trials

Interval estimation of relative risks for combined unilateral and bilateral correlated data

From Pre-test and Post-test Probabilities to Medical Decision Making

Comparison of statistical methods for the analysis of patient-reported outcomes in randomised controlled trials: A simulation study

Unconditional Exact Tests for Equivalence or Noninferiority for Paired Binary Endpoints

Statistical Methods for Comparing Predictive Values in Medical Diagnosis

A Utilitarian Perspective on Risk Quantification for Clinical Significance in Binary Outcomes

A Two-Stage Bayesian Method for Estimating Accuracy and Disease Prevalence for Two Dependent Dichotomous Screening Tests when the Status of Individuals Who Are Negative on Both Tests is Unverified