Abstract:BackgroundThe rank correlation test introduced by Begg and Mazumdar is extensively used in meta-analysis to test for publication bias in clinical and epidemiological studies. It is based on correlating the standardized treatment effect with the variance of the treatment effect using Kendall’s tau as the measure of association. To our knowledge, the operational characteristics regarding the significance level of the test have not, however, been fully assessed.MethodsWe propose an alternative rank correlation test to improve the error rates of the original Begg and Mazumdar test. This test is based on the simulated distribution of the estimated measure of association, conditional on sampling variances. Furthermore, Spearman’s rho is suggested as an alternative rank correlation coefficient. The attained level and power of the tests are studied by simulations of meta-analyses assuming the fixed effects model.ResultsThe significance levels of the original Begg and Mazumdar test often deviate considerably from the nominal level, the null hypothesis being rejected too infrequently. It is proven mathematically that the assumptions for using the rank correlation test are not strictly satisfied. The pairs of variables fail to be independent, and there is a correlation between the standardized effect sizes and sampling variances under the null hypothesis of no publication bias. In the meta-analysis setting, the adverse consequences of a false negative test are more profound than the disadvantages of a false positive test. Our alternative test improves the error rates in fixed effects meta-analysis. Its significance level equals the nominal value, and the Type II error rate is reduced. In small data sets Spearman’s rho should be preferred to Kendall’s tau as the measure of association.ConclusionsAs the attained significance levels of the test introduced by Begg and Mazumdar often deviate greatly from the nominal level, modified rank correlation tests, improving the error rates, should be preferred when testing for publication bias assuming fixed effects meta-analysis.

Some notes on biasedness and unbiasedness of two-sample Kolmogorov-Smirnov test

Kolmogorov–Smirnov Test: Overview

Deriving CDF of Kolmogorov-Smirnov Test Statistic

Differentially Private Kolmogorov-Smirnov-Type Tests

A Higher-Order Kolmogorov-Smirnov Test

On Misuses of the Kolmogorov–Smirnov Test for One-Sample Goodness-of-Fit

How can the score test be consistent?

On the Kolmogorov-Smirnov Test for Normality with Mean and Variance Unknown

The power of visualizing distributional differences: formal graphical n-sample tests

Power comparison of the Kolmogorov–Smirnov test under ranked set sampling and simple random sampling

A Kolmogorov–Smirnov-type test for the two-sample problem with left-truncated data

Improving the error rates of the Begg and Mazumdar test for publication bias in fixed effects meta-analysis

A distance based two-sample test of means difference for multivariate datasets

A Kernel Method for the Two-Sample Problem

The case for balanced hypothesis tests and equal-tailed confidence intervals

Normality and significance testing in simple linear regression model for large sample sizes: a simulation study

Graphical n-sample tests of correspondence of distributions

Two-Sample Smooth Tests for the Equality of Distributions

One and two sample Dvoretzky-Kiefer-Wolfowitz-Massart type inequalities for differing underlying distributions

The two-sample t test: pre-testing its assumptions does not pay off

More accurate tests for the statistical significance of result differences