Abstract:BackgroundMeta-analysis is increasingly used to synthesize proportions (e.g., disease prevalence). It can be implemented with widely used two-step methods or one-step methods, such as generalized linear mixed models (GLMMs). Existing simulation studies have shown that GLMMs outperform the two-step methods in some settings. It is, however, unclear whether these simulation settings are common in the real world. We aim to compare the real-world performance of various meta-analysis methods for synthesizing proportions.MethodsWe extracted datasets of proportions from the Cochrane Library and applied 12 two-step and one-step methods to each dataset. We used Spearman's ρ and the Bland–Altman plot to assess their results' correlation and agreement. The GLMM with the logit link was chosen as the reference method. We calculated the absolute difference and fold change (ratio of estimates) of the overall proportion estimates produced by each method vs. the reference method.ResultsWe obtained a total of 43,644 datasets. The various methods generally had high correlations (ρ > 0.9) and agreements. GLMMs had computational issues more frequently than the two-step methods. However, the two-step methods generally produced large absolute differences from the GLMM with the logit link for small total sample sizes (< 50) and crude event rates within 10–20% and 90–95%, and large fold changes for small total event counts (< 10) and low crude event rates (< 20%).ConclusionsAlthough different methods produced similar overall proportion estimates in most datasets, one-step methods should be considered in the presence of small total event counts or sample sizes and very low or high event rates.

What problem does this paper attempt to address?

This paper aims to solve the problem of performance differences in practical applications among different methods in meta - analysis of synthesizing binary outcome proportions (such as disease prevalence). Specifically, the paper compares the performance of 12 two - step and one - step meta - analysis methods on real - world data sets to evaluate the consistency and differences of these methods when synthesizing proportions. The research background points out that although existing simulation studies show that generalized linear mixed models (GLMMs) are superior to two - step methods in some cases, it is still unclear whether these simulation settings represent the meta - analysis situations in the real world. Therefore, this study uses a large number of meta - analysis data sets in the Cochrane Library to evaluate the performance of different methods in the real world, especially in cases of small sample sizes or extreme event rates. The paper extracts data sets from the Cochrane Library and applies 12 different meta - analysis methods for comparison. The study uses Spearman's rank correlation coefficient \(\rho\) and Bland - Altman plots to evaluate the correlation and consistency of the results of different methods. GLMMs with a log - link are selected as the reference method, and the study calculates the absolute difference and fold change (proportion of estimates) of the overall proportion estimate of each method relative to the reference method. The research results show that various methods generally have high correlation (\(\rho> 0.9\)) and consistency. However, GLMMs encounter more problems during the calculation process than two - step methods, especially when the total sample size is small (< 50) and the original event rate is between 10 - 20% or 90 - 95%. In such cases, two - step methods usually produce larger absolute differences. For cases with a small total number of events (< 10) and a low original event rate (< 20%), two - step methods will produce larger fold changes. The conclusion part points out that although in most data sets, different methods produce similar overall proportion estimates, in cases where there are small total numbers of events or sample sizes and very low or high event rates, one - step methods should be considered. In addition, two - step methods usually produce narrower confidence intervals, which may mean that the coverage probability of their confidence intervals is poor, while one - step GLMMs may have more appropriate confidence intervals because they fully consider the within - study uncertainty.

Empirical Comparisons of 12 Meta-analysis Methods for Synthesizing Proportions of Binary Outcomes

A re-analysis of about 60,000 sparse data meta-analyses suggests that using an adequate method for pooling matters

Laplace approximation, penalized quasi-likelihood, and adaptive Gauss–Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data

A comparison of methods for the analysis of binomial proportion data in behavioral research

Comparison of statistical methods for the analysis of patient-reported outcomes in randomised controlled trials: A simulation study

Bivariate multilevel meta-analysis of log response ratio and standardized mean difference for robust and reproducible environmental and biological sciences

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

Meta‐Analysis of Studies with Missing Data

Empirical Comparison of Publication Bias Tests in Meta-Analysis

Simulation models for aggregated data meta-analysis: Evaluation of pooling effect sizes and publication biases

Meta-Analytical Approaches to Ordinal Outcome Data in Clinical Interventional Studies: A Scoping Review with Reproducible Research

Explaining heterogeneity in meta-analysis: a comparison of methods

Meta‐epidemiological Study: a Step‐by‐step Approach by Using R

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

Mapping between measurement scales in meta‐analysis, with application to measures of body mass index in children

Ratio of means for analyzing continuous outcomes in meta-analysis performed as well as mean difference methods

Meta-analysis of community-based cluster randomization trials with binary outcomes

Evaluation of various estimators for standardized mean difference in meta‐analysis

Robust Point and Variance Estimation for Meta‐analyses with Selective Reporting and Dependent Effect Sizes

Mapping between measurement scales in meta-analysis, with application to measures of body mass index in children

Synthesizing and presenting findings using other methods