Abstract:This study provides a systematic review of the methodological features of meta-analyses in second language learning. The synthesis aims to inform how meta-analyses in L2 learning have been conducted, evaluate whether methodological decisions are aligned with norms and standards, identify issues, and suggest solutions based on expert advice and statistical guides. A total of 120 meta-analyses were retrieved and coded for key features related to bibliographic and demographic characteristics, search and selection, publication bias, quality control, data coding, data analysis, and effect size interpretation. The synthesis showed that 98 meta-analyses examined the effectiveness of instructional treatments, 21 investigated correlations, and one explored the occurrence of events. These meta-analyses included an average of 37 primary studies (range = 9–302). Common selection criteria the meta-analyses applied included publication type, availability of data for effect size calculation, learner traits, learners' target languages, publication dates, publication language, independent variables, dependent variables, and so on. Major strategies used to detect publication bias included creating a funnel plot, using trim-and-fill analysis, and calculating a fail-safe N. Typical moderators examined in the meta-analyses related to research context, treatment features, sample characteristics, and outcome measures. The synthesis also identified a number of issues, including failure to report key features such as model selection (fixed- vs. random-effects model), effect size weighting, and so on; conducting moderator analysis based on very small cell sizes (e.g., only one study in a subgroup); lack of justification for certain methodological decisions such as using d instead of g, using confidence intervals rather than Q-tests to identify significant moderators; lack of quality control; and confounding study-based and synthesis-based moderators. We offer advice on identified issues, and call for more transparency of reporting.

Second-order synthesis of meta-analytic studies in applied linguistics (1998–2021)

A systematic review of meta-analyses in second language research: current practices, issues, and recommendations

Meta-analysis in Second Language Research: Choices and Challenges

Synthesizing and presenting findings using other methods

Towards more scientific meta-analyses

A Call For Cautious Interpretation Of Meta-Analytic Reviews

Longitudinal Analysis of Meta-Analysis Literatures in the Database of ISI Web of Science.

Second language motivation research 2010–2019: a synthetic exploration

Domain definition and search techniques in meta-analyses of L2 research (Or why 18 meta-analyses of feedback have different results)

Robust Point and Variance Estimation for Meta‐analyses with Selective Reporting and Dependent Effect Sizes

Collaboration network of applied linguistics research articles with different methodological orientations

Meta-analysis in information systems research: method choices and recommendations for future research

Context and Implications Document for: Ongoing developments in meta-analytic and quantitative synthesis methods: Broadening the types of research questions that can be addressed

Synthesising results of meta-analyses to inform policy: a comparison of fast-track methods

On the reproducibility of meta-analyses: six practical recommendations

Applying meta-analysis to research on bilingualism: An introduction

Effect Size Reporting Practices in Applied Linguistics Research: A Study of One Major Journal

What do meta-analysts need in primary studies? Guidelines and the SEMI checklist for facilitating cumulative knowledge

A scientometric analysis of applied linguistics research (1970–2022): methodology and future directions

Using machine learning for continuous updating of meta-analysis in educational context

Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences