Abstract:BackgroundThe rank correlation test introduced by Begg and Mazumdar is extensively used in meta-analysis to test for publication bias in clinical and epidemiological studies. It is based on correlating the standardized treatment effect with the variance of the treatment effect using Kendall’s tau as the measure of association. To our knowledge, the operational characteristics regarding the significance level of the test have not, however, been fully assessed.MethodsWe propose an alternative rank correlation test to improve the error rates of the original Begg and Mazumdar test. This test is based on the simulated distribution of the estimated measure of association, conditional on sampling variances. Furthermore, Spearman’s rho is suggested as an alternative rank correlation coefficient. The attained level and power of the tests are studied by simulations of meta-analyses assuming the fixed effects model.ResultsThe significance levels of the original Begg and Mazumdar test often deviate considerably from the nominal level, the null hypothesis being rejected too infrequently. It is proven mathematically that the assumptions for using the rank correlation test are not strictly satisfied. The pairs of variables fail to be independent, and there is a correlation between the standardized effect sizes and sampling variances under the null hypothesis of no publication bias. In the meta-analysis setting, the adverse consequences of a false negative test are more profound than the disadvantages of a false positive test. Our alternative test improves the error rates in fixed effects meta-analysis. Its significance level equals the nominal value, and the Type II error rate is reduced. In small data sets Spearman’s rho should be preferred to Kendall’s tau as the measure of association.ConclusionsAs the attained significance levels of the test introduced by Begg and Mazumdar often deviate greatly from the nominal level, modified rank correlation tests, improving the error rates, should be preferred when testing for publication bias assuming fixed effects meta-analysis.

Karl Pearson's meta-analysis revisited

EBT: a Statistic Test Identifying Moderate Size of Significant Features with Balanced Power and Precision for Genome-Wide Rate Comparisons

Power Analysis of Principal Components Regression in Genetic Association Studies.

Decomposing Pearson's Χ2 Test: A Linear Regression and Its Departure from Linearity

Improving the error rates of the Begg and Mazumdar test for publication bias in fixed effects meta-analysis

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

Empirically adjusted fixed-effects meta-analysis methods in genomic studies

Bagged Empirical Null p-values: A Method to Account for Model Uncertainty in Large Scale Inference

PASTRY: achieving balanced power for detecting risk and protective minor alleles in meta-analysis of association studies with overlapping subjects

High-Dimensional Randomized Crossover Studies: A Clarification of P-Values Interpretation

High-dimensional Randomization-Based Inference Capitalizing on Classical Design and Modern Computing

Karl Pearson and the Logic of Science: Renouncing Causal Understanding (the Bride) and Inverted Spinozism

Comparison of Three Major Meta-Analytic Approaches.

On the P-value for Members of the Cressie-Read Family of Divergence Statistics

Higher-Order Asymptotics and Its Application to Testing the Equality of the Examinee Ability Over Two Sets of Items

Computationally efficient meta-analysis of gene-based tests using summary statistics in large-scale genetic studies

Operating Characteristics of a Rank Correlation Test for Publication Bias

COMPARISON OF 3 MAJOR METAANALYTIC APPROACHES

Valid and Informative P-Values from Big Data, Illustrated in Environmental Epigenomics

Comparing Correlated but Nonoverlapping Correlations.

P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods