Abstract:This study provides a systematic review of the methodological features of meta-analyses in second language learning. The synthesis aims to inform how meta-analyses in L2 learning have been conducted, evaluate whether methodological decisions are aligned with norms and standards, identify issues, and suggest solutions based on expert advice and statistical guides. A total of 120 meta-analyses were retrieved and coded for key features related to bibliographic and demographic characteristics, search and selection, publication bias, quality control, data coding, data analysis, and effect size interpretation. The synthesis showed that 98 meta-analyses examined the effectiveness of instructional treatments, 21 investigated correlations, and one explored the occurrence of events. These meta-analyses included an average of 37 primary studies (range = 9–302). Common selection criteria the meta-analyses applied included publication type, availability of data for effect size calculation, learner traits, learners' target languages, publication dates, publication language, independent variables, dependent variables, and so on. Major strategies used to detect publication bias included creating a funnel plot, using trim-and-fill analysis, and calculating a fail-safe N. Typical moderators examined in the meta-analyses related to research context, treatment features, sample characteristics, and outcome measures. The synthesis also identified a number of issues, including failure to report key features such as model selection (fixed- vs. random-effects model), effect size weighting, and so on; conducting moderator analysis based on very small cell sizes (e.g., only one study in a subgroup); lack of justification for certain methodological decisions such as using d instead of g, using confidence intervals rather than Q-tests to identify significant moderators; lack of quality control; and confounding study-based and synthesis-based moderators. We offer advice on identified issues, and call for more transparency of reporting.

Fragile Findings, Complex Conclusions, and Meta-Analysis in Special Education

Meta-analyses as a privileged information source for informing teachers' practice?

Publication Bias in Special Education Meta-Analyses

A Call For Cautious Interpretation Of Meta-Analytic Reviews

Single-case design meta-analyses in education and psychology: a systematic review of methodology

A systematic review of meta-analyses in second language research: current practices, issues, and recommendations

Meta-analysis in Second Language Research: Choices and Challenges

Synthesizing and presenting findings using other methods

Meta-Analysis With Complex Research Designs: Dealing With Dependence From Multiple Measures and Multiple Group Comparisons

Technology-Aided Instruction and Intervention for Students With ASD: A Meta-Analysis Using Novel Methods of Estimating Effect Sizes for Single-Case Research

A Systematic Review of Meta-Analyses on the Impact of Formative Assessment on K-12 Students' Learning: Toward Sustainable Quality Education

Fail, flip, fix, and feed – Rethinking flipped learning: A review of meta-analyses and a subsequent meta-analysis

Applying meta-analysis to research on bilingualism: An introduction

Qualitative Methods in Special Education Research

Synthesizing Results From Empirical Research on Computer-Based Scaffolding in STEM Education: A Meta-Analysis

Interpretation of data from multiple trials: a critical review

When meta-analysis misleads: A critical case study of a meta-analysis of client feedback.

Exploring the fragility of meta-analyses in ophthalmology: a systematic review

Interpretation of meta-analyses

Applications of Meta-Analysis to Library and Information Science Research: Content Analysis

Prevalence of Research Design in Special Education: A Survey of Peer-Reviewed Journals