Abstract:INTRODUCTION: After several decades' development, meta-analysis has become the pillar of evidence-based medicine. However, heterogeneity is still the threat to the validity and quality of such studies. Currently, Q and its descendant I(2) (I square) tests are widely used as the tools for heterogeneity evaluation. The core mission of this kind of test is to identify data sets from similar populations and exclude those are from different populations. Although Q and I(2) are used as the default tool for heterogeneity testing, the work we present here demonstrates that the robustness of these two tools is questionable.METHODS AND FINDINGS: We simulated a strictly normalized population S. The simulation successfully represents randomized control trial data sets, which fits perfectly with the theoretical distribution (experimental group: p = 0.37, control group: p = 0.88). And we randomly generate research samples Si that fits the population with tiny distributions. In short, these data sets are perfect and can be seen as completely homogeneous data from the exactly same population. If Q and I(2) are truly robust tools, the Q and I(2) testing results on our simulated data sets should not be positive. We then synthesized these trials by using fixed model. Pooled results indicated that the mean difference (MD) corresponds highly with the true values, and the 95% confidence interval (CI) is narrow. But, when the number of trials and sample size of trials enrolled in the meta-analysis are substantially increased; the Q and I(2) values also increase steadily. This result indicates that I(2) and Q are only suitable for testing heterogeneity amongst small sample size trials, and are not adoptable when the sample sizes and the number of trials increase substantially.CONCLUSIONS: Every day, meta-analysis studies which contain flawed data analysis are emerging and passed on to clinical practitioners as "updated evidence". Using this kind of evidence that contain heterogeneous data sets leads to wrong conclusion, makes chaos in clinical practice and weakens the foundation of evidence-based medicine. We suggest more strict applications of meta-analysis: it should only be applied to those synthesized trials with small sample sizes. We call upon that the tools of evidence-based medicine should keep up-to-dated with the cutting-edge technologies in data science. Clinical research data should be made available publicly when there is any relevant article published so the research community could conduct in-depth data mining, which is a better alternative for meta-analysis in many instances.

Investigating the Heterogeneity of "Study Twins"

Meta-analysis of two studies in the presence of heterogeneity with applications in rare diseases

Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies

Dynamically borrowing strength from another study through shrinkage estimation

Explaining heterogeneity in meta-analysis: a comparison of methods

Meta‐Analysis of Studies with Missing Data

Comparison of methods that combine multiple randomized trials to estimate heterogeneous treatment effects

Analysis of 1:1 Matched Cohort Studies and Twin Studies, with Binary Exposures and Binary Outcomes

Lifting the veil off treatment effect heterogeneity

Familial confounding or measurement error? How to interpret findings from sibling and co-twin control studies

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

Meta-analysis of few small studies in orphan diseases

Statistical Primer: heterogeneity, random- or fixed-effects model analyses?

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

A re-analysis of about 60,000 sparse data meta-analyses suggests that using an adequate method for pooling matters

Meta‐epidemiological Study: a Step‐by‐step Approach by Using R

A Two-Stage Method for Extending Inferences from a Collection of Trials

Joint meta-analysis of two diagnostic tests accounting for within and between studies dependence

The Dilemma of Heterogeneity Tests in Meta-Analysis: A Challenge from a Simulation Study

Meta-analysis of community-based cluster randomization trials with binary outcomes

Beta-binomial meta-analysis of individual differences based on sample means and standard deviations: Studying reliability of sum scores of binary items.