A systematic review of meta-analyses in second language research: current practices, issues, and recommendations
Alyssa Vuogan,Shaofeng Li
DOI: https://doi.org/10.1515/applirev-2022-0192
IF: 3.063
2023-01-11
Applied Linguistics Review
Abstract:This study provides a systematic review of the methodological features of meta-analyses in second language learning. The synthesis aims to inform how meta-analyses in L2 learning have been conducted, evaluate whether methodological decisions are aligned with norms and standards, identify issues, and suggest solutions based on expert advice and statistical guides. A total of 120 meta-analyses were retrieved and coded for key features related to bibliographic and demographic characteristics, search and selection, publication bias, quality control, data coding, data analysis, and effect size interpretation. The synthesis showed that 98 meta-analyses examined the effectiveness of instructional treatments, 21 investigated correlations, and one explored the occurrence of events. These meta-analyses included an average of 37 primary studies (range = 9–302). Common selection criteria the meta-analyses applied included publication type, availability of data for effect size calculation, learner traits, learners' target languages, publication dates, publication language, independent variables, dependent variables, and so on. Major strategies used to detect publication bias included creating a funnel plot, using trim-and-fill analysis, and calculating a fail-safe N. Typical moderators examined in the meta-analyses related to research context, treatment features, sample characteristics, and outcome measures. The synthesis also identified a number of issues, including failure to report key features such as model selection (fixed- vs. random-effects model), effect size weighting, and so on; conducting moderator analysis based on very small cell sizes (e.g., only one study in a subgroup); lack of justification for certain methodological decisions such as using d instead of g, using confidence intervals rather than Q-tests to identify significant moderators; lack of quality control; and confounding study-based and synthesis-based moderators. We offer advice on identified issues, and call for more transparency of reporting.
linguistics