Abstract:Abstract Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results. We present a comparative analysis of time‐synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms. We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions. To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine‐readable search metadata.

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Replication research in second language acquisition studies

Replication studies in the Netherlands: Lessons learned and recommendations for funders, publishers and editors, and universities

Irreproducibility in Searches of Scientific Literature: A Comparative Analysis.

Are replications mainstream now? A comparison of support for replications expressed in the policies of social psychology journals in 2015 and 2022

Replication of Special Education Research

Play It Again, Sam! An Analysis of Constructive Replication in the Organizational Sciences

Replication is important for educational psychology: Recent developments and key issues

Replication Studies Using Secondary or Nonexperimental Datasets

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication

You Can't Publish Replication Studies (and How to Anyways)

A Systematic Review on Reproducibility in Child-Robot Interaction

Replication Research and Special Education

Replication studies in international business

Replication of Audit and Financial Accounting Research: We Do More than We Think

Recommendations for Replication Research in Special Education

Interdisciplinary lessons and recommendations for the evaluation of replicability in behavioral sciences

Replication and Reproducibility in Cross-Cultural Psychology

Promoting computationally reproducible research in applied linguistics: Recommended practices and considerations

The Language of (Non)Replicable Social Science