Abstract:Abstract Background The replication crisis hit the medical sciences about a decade ago, but today still most of the flaws inherent in null hypothesis significance testing (NHST) have not been solved. While the drawbacks of p -values have been detailed in endless venues, for clinical research, only a few attractive alternatives have been proposed to replace p -values and NHST. Bayesian methods are one of them, and they are gaining increasing attention in medical research, as some of their advantages include the description of model parameters in terms of probability, as well as the incorporation of prior information in contrast to the frequentist framework. While Bayesian methods are not the only remedy to the situation, there is an increasing agreement that they are an essential way to avoid common misconceptions and false interpretation of study results. The requirements necessary for applying Bayesian statistics have transitioned from detailed programming knowledge into simple point-and-click programs like JASP. Still, the multitude of Bayesian significance and effect measures which contrast the gold standard of significance in medical research, the p -value, causes a lack of agreement on which measure to report. Methods Therefore, in this paper, we conduct an extensive simulation study to compare common Bayesian significance and effect measures which can be obtained from a posterior distribution. In it, we analyse the behaviour of these measures for one of the most important statistical procedures in medical research and in particular clinical trials, the two-sample Student’s (and Welch’s) t-test. Results The results show that some measures cannot state evidence for both the null and the alternative. While the different indices behave similarly regarding increasing sample size and noise, the prior modelling influences the obtained results and extreme priors allow for cherry-picking similar to p-hacking in the frequentist paradigm. The indices behave quite differently regarding their ability to control the type I error rates and regarding their ability to detect an existing effect. Conclusion Based on the results, two of the commonly used indices can be recommended for more widespread use in clinical and biomedical research, as they improve the type I error control compared to the classic two-sample t-test and enjoy multiple other desirable properties.

Small sample sizes: A big data problem in high-dimensional data analysis

Tests for a Multiple-Sample Problem in High Dimensions

Finite sample t-tests for high-dimensional means

High-dimensional Randomization-Based Inference Capitalizing on Classical Design and Modern Computing

t-tests, non-parametric tests, and large studies—a paradox of statistical practice?

Sample size and statistical power considerations in high-dimensionality data settings: a comparative study of classification algorithms

A simple statistical framework for small sample studies

Minor Issues Escalated to Critical Levels in Large Samples: A Permutation-Based Fix

Nonparametric High-Dimensional Multi-Sample Tests based on Graph Theory

Multiple multi-sample testing under arbitrary covariance dependency

Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research

Simulation-Based Hypothesis Testing of High Dimensional Means Under Covariance Heterogeneity

Large-Scale Simultaneous Testing Using Kernel Density Estimation

Statistical inference for diagnostic test accuracy studies with multiple comparisons

A More Powerful Two-Sample Test in High Dimensions using Random Projection

Two-sample high dimensional mean test based on prepivots

Hypothesis testing at the extremes: fast and robust association for high-throughput data

Simultaneous critical values for $t$-tests in very high dimensions

Bayesian Optimal Two-sample Tests in High-dimension

ISSUES IN BIOMEDICAL STATISTICS: COMPARING MEANS UNDER NORMAL DISTRIBUTION THEORY

Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method