Abstract:Introduction Meta-analysis cannot proceed unless each study outcome is on the same metric and has an appropriate sampling variance estimate, the inverse of which is used as the weight in meta-analytic statistics. When comparing treatments for trials that use the same continuous measures across studies, contemporary meta-analytic practice uses the unstandardized mean difference (UMD) to model the difference between the observed means (i.e., ME-MC) rather than representing effects in the standardized mean difference (SMD). A fundamental difference between the two strategies is that the UMD incorporates the observed variance of the measures as a component of the analytical weights (viz., sampling error or inverse variance) in statistically modeling the results for each study. In contrast, the SMD incorporates the measure’s variance directly in the effect size itself (i.e., SMD=[ME−MC]/SD) and not directly in the analytical weights. The UMD approach has been conventional even though its bias and efficiency are unknown; these have also not been compared with the SMD. Also unresolved is which of many possible available equations best optimize statistical modeling for the SMD in use with repeated measures designs (one or two groups). Methods Monte Carlo simulations compared available equations in terms of their bias and efficiency across the many different conditions established by crossing: (1) number of studies in the meta-analysis (k = 10, 20, 50, and 100); (2) mean study sample sizes (5 values of N ranging from small to very large); (3) the ratio of the within-study observed measure variances for experimental and control groups and at pretest and post-test (ratios: 1:1, 2:1, and 4:1); (4) the post-test mean of each pseudo experimental group to achieve 3 parametric effect sizes (δ= 0.25, 0.50, and 0.80); (5) normal versus nonnormal distributions (4 levels); and (6) the between-studies variance (τ2= 0, 0.04, 0.08, 0.16, and 0.32). For the second issue, (7) the correlation between the two conditions was manipulated (ρpre-post = 0, 0.25, 0.50, and 0.75). Results and Conclusions This investigation provides guidance for statistical practice in relation to meta-analysis of studies that compare two groups at one point in time, or that examine repeated measures for one or two groups. Simulations showed that neither standardized or unstandardized effect size indexes had an advantage in terms of bias or efficiency when distributions are normal, when there is no heterogeneity among effects, and when the observed variances of the experimental and control groups are equal. In contrast, when conditions deviate from these ideals, the SMD yields better statistical inferences than UMDs in terms of bias and efficiency. Under high skewness and kurtosis, neither metric has a marked advantage. In general, the standardized index presents the least bias under most conditions and is more efficient than the unstandardized index. Finally, the results comparing estimations of the SMD and its variance suggest that some are preferable to others under certain conditions. The current results imply that the choice of effect size metrics, estimators, and sampling variances can have substantial impact on statistical inferences even under such commonly observed circumstances as normal sampling distributions, large numbers of studies, and studies with large samples, and when effects exhibit heterogeneity. Although using the SMD may make clinical inferences more difficult, use of the SMD does permit inferences about effect size magnitude. The Discussion considers clinical interpretation of results using the SMD and addresses limitations of the current project.

Choosing effect measures and computing estimates of effect

Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs

Meta‐epidemiological Study: a Step‐by‐step Approach by Using R

The Measurement and Communication of Effect Sizes in Management Research

A Comparison of Single-Case Effect Measures Using Check-In Check-Out Data

Defining and Estimating Intervention Effects for Groups that will Develop an Auxiliary Outcome

Meta-Analytic Statistical Inferences for Continuous Measure Outcomes As a Function of Effect Size Metric and Other Assumptions

Statistical considerations for outcomes in clinical research: A review of common data types and methodology

Effect Size Estimation for One-Sample Multiple-Choice-type Data: Design, Analysis, and Meta-Analysis.

Understanding effect size: an international online survey among psychiatrists, psychologists, physicians from other medical specialities, dentists and other health professionals

Efficiency Study of Estimators for a Treatment Effect in a Pretest–Posttest Trial

Efficient Study Design with Multiple Measurement Instruments

A brief note on the common (fixed)-effect meta-analysis model

A practical guide to estimating treatment effects in patients with rheumatic diseases using real-world data

An Evidence-Based Approach to Choosing Outcome Measures: A Checklist for the Critical Appraisal of Validity, Reliability and Responsiveness Studies

Testing Whether Reported Treatment Effects are Unduly Dependent on the Specific Outcome Measure Used

Calculating confidence intervals for impact numbers

Using simulation studies to evaluate statistical methods

Defining the Primary Outcomes and Justifying Secondary Outcomes of a Study: Usually, the Fewer, the Better

Introduction to Connonly Used Effect Sizes in Meta-analysis

Power calculation for causal inference in social science: sample size and minimum detectable effect determination