Abstract:Experimental studies in biomedical research frequently pose analytical problems related to small sample size. In such studies, there are conflicting findings regarding the choice of parametric and nonparametric analysis, especially with non-normal data. In such instances, some methodologists questioned the validity of parametric tests and suggested nonparametric tests. In contrast, other methodologists found nonparametric tests to be too conservative and less powerful and thus preferred using parametric tests. Some researchers have recommended using a bootstrap test; however, this method also has small sample size limitation. We used a pooled method in nonparametric bootstrap test that may overcome the problem related with small samples in hypothesis testing. The present study compared nonparametric bootstrap test with pooled resampling method corresponding to parametric, nonparametric, and permutation tests through extensive simulations under various conditions and using real data examples. The nonparametric pooled bootstrap t-test provided equal or greater power for comparing two means as compared with unpaired t-test, Welch t-test, Wilcoxon rank sum test, and permutation test while maintaining type I error probability for any conditions except for Cauchy and extreme variable lognormal distributions. In such cases, we suggest using an exact Wilcoxon rank sum test. Nonparametric bootstrap paired t-test also provided better performance than other alternatives. Nonparametric bootstrap test provided benefit over exact Kruskal-Wallis test. We suggest using nonparametric bootstrap test with pooled resampling method for comparing paired or unpaired means and for validating the one way analysis of variance test results for non-normal data in small sample size studies. Copyright © 2017 John Wiley & Sons, Ltd.

Bootstrap Matching: a robust and efficient correction for non-random A/B test, and its applications

Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

Testing Biased Randomization Assumptions and Quantifying Imperfect Matching and Residual Confounding in Matched Observational Studies

Treatment bootstrapping: A new approach to quantify uncertainty of average treatment effect estimates

Rapid and Scalable Bayesian AB Testing

Cluster-Adaptive Network A/B Testing: From Randomization to Estimation

Validation of massively-parallel adaptive testing using dynamic control matching

Bootstrapping Inference of Average Treatment Effect in Completely Randomized Experiments with High-Dimensional Covariates

A matching design for augmenting a randomized clinical trial with external control

Matching Bounds: How Choice of Matching Algorithm Impacts Treatment Effects Estimates and What to Do about It

Approximate Bayesian Bootstrap procedures to estimate multilevel treatment effects in observational studies with application to type 2 diabetes treatment regimens

Better Experimental Design by Hybridizing Binary Matching with Imbalance Optimization

The Big Data Bootstrap

An Online Sequential Test for Qualitative Treatment Effects

Adaptive A/B Tests and Simultaneous Treatment Parameter Optimization

Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method

Optimality of Matched-Pair Designs in Randomized Controlled Trials

Multiple Comparisons of Treatment Against Control under Unequal Variances Using Parametric Bootstrap

Empirical Bayes Multistage Testing for Large-Scale Experiments

Randomization Tests for Adaptively Collected Data

Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework