Effects of prostaglandin B2 on vascular tone and reactivity.

S. Greenberg,F. Diecke,P. Kadowitz,J. P. Long

DOI: https://doi.org/10.1016/0024-3205(75)90257-X

1975-02-01

Life Science

Abstract:

What problem does this paper attempt to address?

Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Michael Gusenbauer,Neal R. Haddaway

DOI: https://doi.org/10.1002/jrsm.1378

2020-01-28

Research Synthesis Methods

Abstract:<p>Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses), because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments.</p><p>This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analysed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system.</p><p>We call for database owners to recognise the requirements of evidence synthesis, and for academic journals to re‐assess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.</p>

multidisciplinary sciences,mathematical & computational biology
Irreproducibility in Searches of Scientific Literature: A Comparative Analysis.

Gabor Pozsgai,Gabor L. Lovei,Liette Vasseur,Geoff Gurr,Peter Batary,Janos Korponai,Nick A. Littlewood,Jian Liu,Arnold Mora,John Obrycki,Olivia Reynolds,Jenni A. Stockan,Heather VanVolkenburg,Jie Zhang,Wenwu Zhou,Minsheng You

DOI: https://doi.org/10.1002/ece3.8154

2020-01-01

Abstract:Abstract Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results. We present a comparative analysis of time‐synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms. We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions. To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine‐readable search metadata.
Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Wichor M. Bramer,Melissa L. Rethlefsen,Jos Kleijnen,Oscar H. Franco

DOI: https://doi.org/10.1186/s13643-017-0644-y

2017-12-01

Systematic Reviews

Abstract:BackgroundWithin systematic reviews, when searching for relevant references, it is advisable to use multiple databases. However, searching databases is laborious and time-consuming, as syntax of search strategies are database specific. We aimed to determine the optimal combination of databases needed to conduct efficient searches in systematic reviews and whether the current practice in published reviews is appropriate. While previous studies determined the coverage of databases, we analyzed the actual retrieval from the original searches for systematic reviews.MethodsSince May 2013, the first author prospectively recorded results from systematic review searches that he performed at his institution. PubMed was used to identify systematic reviews published using our search strategy results. For each published systematic review, we extracted the references of the included studies. Using the prospectively recorded results and the studies included in the publications, we calculated recall, precision, and number needed to read for single databases and databases in combination. We assessed the frequency at which databases and combinations would achieve varying levels of recall (i.e., 95%). For a sample of 200 recently published systematic reviews, we calculated how many had used enough databases to ensure 95% recall.ResultsA total of 58 published systematic reviews were included, totaling 1746 relevant references identified by our database searches, while 84 included references had been retrieved by other search methods. Sixteen percent of the included references (291 articles) were only found in a single database; Embase produced the most unique references (n = 132). The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar performed best, achieving an overall recall of 98.3 and 100% recall in 72% of systematic reviews. We estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references as many fail to search important databases. Other specialized databases, such as CINAHL or PsycINFO, add unique references to some reviews where the topic of the review is related to the focus of the database.ConclusionsOptimal searches in systematic reviews should search at least Embase, MEDLINE, Web of Science, and Google Scholar as a minimum requirement to guarantee adequate and efficient coverage.

medicine, general & internal
Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices

Michael Gusenbauer

DOI: https://doi.org/10.1002/jrsm.1729

2024-06-16

Research Synthesis Methods

Abstract:Citation indices providing information on backward citation (BWC) and forward citation (FWC) links are essential for literature discovery, bibliographic analysis, and knowledge synthesis, especially when language barriers impede document identification. However, the suitability of citation indices varies. While some have been analyzed, the majority, whether new or established, lack comprehensive evaluation. Therefore, this study evaluates the citation coverage of the citation indices of 59 databases, encompassing the widely used Google Scholar, Scopus, and Web of Science alongside many others never previously analyzed, such as the emerging Lens, Scite, Dimensions, and OpenAlex or the subject‐specific PubMed and JSTOR. Through a comprehensive analysis using 259 journal articles from across disciplines, this research aims to guide scholars in selecting indices with broader document coverage and more accurate and comprehensive backward and forward citation links. Key findings highlight Google Scholar, ResearchGate, Semantic Scholar, and Lens as leading options for FWC searching, with Lens providing superior download capabilities. For BWC searching, the Web of Science Core Collection can be recommended over Scopus for accuracy. BWC information from publisher databases such as IEEE Xplore or ScienceDirect was generally found to be the most accurate, yet only available for a limited number of articles. The findings will help scholars conducting systematic reviews, meta‐analyses, and bibliometric analyses to select the most suitable databases for citation searching.

multidisciplinary sciences,mathematical & computational biology
The optimal approach for retrieving systematic reviews was achieved when searching MEDLINE and Epistemonikos in addition to reference checking: a methodological validation study

Lena Heinen,Käthe Goossen,Carole Lunny,Julian Hirt,Livia Puljak,Dawid Pieper

DOI: https://doi.org/10.1186/s12874-024-02384-2

2024-11-10

BMC Medical Research Methodology

Abstract:Systematic reviews (SRs) are used to inform clinical practice guidelines and healthcare decision making by synthesising the results of primary studies. Efficiently retrieving as many relevant SRs as possible is challenging with a minimum number of databases, as there is currently no guidance on how to do this optimally. In a previous study, we determined which individual databases contain the most SRs, and which combination of databases retrieved the most SRs. In this study, we aimed to validate those previous results by using a different, larger, and more recent set of SRs.

health care sciences & services
Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses

Matthew E. Falagas,Eleni I. Pitsouni,George A. Malietzis,Georgios Pappas

DOI: https://doi.org/10.1096/fj.07-9492lsf

2007-09-20

The FASEB Journal

Abstract:The evolution of the electronic age has led to the development of numerous medical databases on the World Wide Web, offering search facilities on a particular subject and the ability to perform citation analysis. We compared the content coverage and practical utility of PubMed, Scopus, Web of Science, and Google Scholar. The official Web pages of the databases were used to extract information on the range of journals covered, search facilities and restrictions, and update frequency. We used the example of a keyword search to evaluate the usefulness of these databases in biomedical information retrieval and a specific published article to evaluate their utility in performing citation analysis. All databases were practical in use and offered numerous search facilities. PubMed and Google Scholar are accessed for free. The keyword search with PubMed offers optimal update frequency and includes online early articles; other databases can rate articles by number of citations, as an index of importance. For citation analysis, Scopus offers about 20% more coverage than Web of Science, whereas Google Scholar offers results of inconsistent accuracy. PubMed remains an optimal tool in biomedical electronic research. Scopus covers a wider journal range, of help both in keyword searching and citation analysis, but it is currently limited to recent articles (published after 1995) compared with Web of Science. Google Scholar, as for the Web in general, can help in the retrieval of even the most obscure information but its use is marred by inadequate, less often updated, citation information.
Where you search determines what you find: the effects of bibliographic databases on systematic reviews

Seperia B. Wanyama,Ronald W. McQuaid,Markus Kittler

DOI: https://doi.org/10.1080/13645579.2021.1892378

2021-03-05

International Journal of Social Research Methodology

Abstract:Systematic literature reviews are common in social research for integrating and synthesising existing research. This paper argues that the outcomes of such reviews are affected by the choice of bibliographic databases. It presents evidence of substantial variation across three large electronic databases (Scopus, Web of Science and EBSCO) in a study on employee retention and staff turnover. It considers the specific articles, numbers returned, numbers shared across databases and perceived quality of journals hosting the retrieved articles. Results show that only 130 articles (5.7% of 2267 retrieved) were found common to all three databases, suggesting that decisions on how and where literature is retrieved can substantially affect the results of systematic reviews and meta-analyses. The findings caution against the use of single databases and claiming comprehensiveness. The paper reflects on how additional literature search methods (e.g., contacting experts, citation indices) and their sequence of use can affect systematic review quality.

social sciences, interdisciplinary
Systematically Searching Empirical Literature in the Social Sciences: Results from Two Meta-Analyses Within the Domain of Education

David I. Pickup,Robert M. Bernard,Eugene Borokhovski,Anne C. Wade,Rana M. Tamim

DOI: https://doi.org/10.21702/rpj.2018.4.10

2019-01-31

Российский психологический журнал

Abstract:Introduction. This paper provides an overview of the information retrieval strategy employed for two meta-analyses, conducted by a systematic review team at Concordia University (Montreal, QC, Canada). Both papers draw on standards first articulated by H.M. Cooper and further developed by the Campbell Collaboration, which promote a comprehensive approach to systematically searching an extensive array of resources (bibliographic databases, print resources, citation indices, etc.) in order to locate both published and unpublished research. The goal is to verify if searching comprehensively through multiple resources retrieves studies that are unique, and hence, improve the overall representativeness of a diverse body of literature. We also analyze the sensitivity and specificity of the results by data source. Methods. In order to determine the source sensitivity, we consider percentage of results from each source retrieved for full-text review. In order to determine the source specificity, we derive a percentage from the total number of studies included in the final meta-analysis compared against the overall number of initial results found. Results. Results demonstrate the need to search beyond the subject-specific databases of a particular discipline as unique results can be found in many places. Databases for related disciplines provided 129 unique includes to each meta-analysis, and multidisciplinary databases provided 44 and 99 unique includes for the two meta-analyses in question respectively. Manual search techniques were much more sensitive and specific than electronic searches of databases and yield a higher percentage of final includes. Discussion. The results demonstrate the utility of a comprehensive information retrieval methodology like that proposed by the Campbell Collaboration, which goes beyond the main subject databases to locate the full range of information sources, including grey literature.
The contribution of databases to the results of systematic reviews: a cross-sectional study

Lisa Hartling,Robin Featherstone,Megan Nuspl,Kassi Shave,Donna M. Dryden,Ben Vandermeer

DOI: https://doi.org/10.1186/s12874-016-0232-1

2016-09-26

BMC Medical Research Methodology

Abstract:BackgroundOne of the best sources for high quality information about healthcare interventions is a systematic review. A well-conducted systematic review includes a comprehensive literature search. There is limited empiric evidence to guide the extent of searching, in particular the number of electronic databases that should be searched. We conducted a cross-sectional quantitative analysis to examine the potential impact of selective database searching on results of meta-analyses.MethodsOur sample included systematic reviews (SRs) with at least one meta-analysis from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of relevant studies indexed in each of 10 databases; and 2) changes in results and statistical significance of primary meta-analysis for studies identified in Medline only and in Medline plus each of the other databases.ResultsDue to variation across topics, we present results by group (ARI n = 57, ID n = 38, DPLP n = 34). For ARI, identification of relevant studies was highest for Medline (85 %) and Embase (80 %). Restricting meta-analyses to trials that appeared in Medline + Embase yielded fewest changes in statistical significance: 53/55 meta-analyses showed no change. Point estimates changed in 12 cases; in 7 the change was less than 20 %. For ID, yield was highest for Medline (92 %), Embase (81 %), and BIOSIS (67 %). Restricting meta-analyses to trials that appeared in Medline + BIOSIS yielded fewest changes with 1 meta-analysis changing in statistical significance. Point estimates changed in 8 of 31 meta-analyses; change less than 20 % in all cases. For DPLP, identification of relevant studies was highest for Medline (75 %) and Embase (62 %). Restricting meta-analyses to trials that appeared in Medline + PsycINFO resulted in only one change in significance. Point estimates changed for 13 of 33 meta-analyses; less than 20 % in 9 cases.ConclusionsMajority of relevant studies can be found within a limited number of databases. Results of meta-analyses based on the majority of studies did not differ in most cases. There were very few cases of changes in statistical significance. Effect estimates changed in a minority of meta-analyses but in most the change was small. Results did not change in a systematic manner (i.e., regularly over- or underestimating treatment effects), suggesting that selective searching may not introduce bias in terms of effect estimates.

health care sciences & services
Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies

Claes Wohlin,Marcos Kalinowski,Katia Romero Felizardo,Emilia Mendes

DOI: https://doi.org/10.1016/j.infsof.2022.106908

2023-07-06

Abstract:Background: A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which has later been complemented with snowball searches. Our conjecture is that we can perform even better searches if combining the two search approaches, referred to as a hybrid search strategy. Objective: Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared some alternative hybrid search strategies to assess whether it was possible to identify more cost-efficient ways of searching for relevant primary studies. Method: To compare and evaluate the hybrid search strategy, we replicated an SLR on industry-academia collaboration in software engineering. The SLR used a more traditional approach to searching for relevant articles for an SLR, while the replication was conducted using a hybrid search strategy. Results: In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30 percent more primary studies and even more when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when conducting systematic literature studies. Conclusions: The hybrid search strategy is a strong contender for being used when conducting systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.

Software Engineering
The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

Neal Robert Haddaway,Alexandra Mary Collins,Deborah Coughlin,Stuart Kirk

DOI: https://doi.org/10.1371/journal.pone.0138237

IF: 3.7

2015-09-17

PLoS ONE

Abstract:Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10-67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations' websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

multidisciplinary sciences
A systematic approach to searching: an efficient and complete method to develop literature searches

Wichor M. Bramer,Gerdien B. de Jonge,Melissa L. Rethlefsen,Frans Mast,Jos Kleijnen,Gerdien B. De Jonge

DOI: https://doi.org/10.5195/jmla.2018.283

2018-10-04

Abstract:Creating search strategies for systematic reviews, finding the best balance between sensitivity and specificity, and translating search strategies between databases is challenging. Several methods describe standards for systematic search strategies, but a consistent approach for creating an exhaustive search strategy has not yet been fully described in enough detail to be fully replicable. The authors have established a method that describes step by step the process of developing a systematic search strategy as needed in the systematic review. This method describes how single-line search strategies can be prepared in a text document by typing search syntax (such as field codes, parentheses, and Boolean operators) before copying and pasting search terms (keywords and free-text synonyms) that are found in the thesaurus. To help ensure term completeness, we developed a novel optimization technique that is mainly based on comparing the results retrieved by thesaurus terms with those retrieved by the free-text search words to identify potentially relevant candidate search terms. Macros in Microsoft Word have been developed to convert syntaxes between databases and interfaces almost automatically. This method helps information specialists in developing librarian-mediated searches for systematic reviews as well as medical and health care practitioners who are searching for evidence to answer clinical questions. The described method can be used to create complex and comprehensive search strategies for different databases and interfaces, such as those that are needed when searching for relevant references for systematic reviews, and will assist both information specialists and practitioners when they are searching the biomedical literature.

information science & library science
158: ASSESSING THE QUALITY OF SEARCH STRATEGIES IN DECISION SUPPORT SYSTEM (DSS) SYSTEMATIC REVIEWS

Zahra Fathifar,Hossein Saeedi Hasankandi,Fatemeh Sadeghi-Ghyassi

DOI: https://doi.org/10.1136/bmjopen-2016-015415.158

IF: 3.006

2017-02-01

BMJ Open

Abstract:Background and aims: Search strategies play a major role in developing a systematic review. A comprehensive literature search dramatically reduces publication bias. The higher sensitive search leads to a more well-prepared systematic review. The aim of this study is evaluating the quality of search strategies in decision support system (DSS) systematic reviews. Methods: A comprehensive literature search using MEDLINE (via Ovid), Embase, Pubmed and Cochrane Library was conducted until September August 2016. The search was limited to systematic reviews and with no language restriction. Meta-reviews were excluded. Only results of searching for “decision support systems” or “decision support system” keywords and limited to title of documents were included in this study. Two independent authors screened the results. Disagreement was resolved by discussion. The search strategies of the included studies were assessed with a nine-question checklist prepared by Faggion et al 1 . Data were extracted into Excel. Results: A total of 47 systematic reviews (SRs) were included. There was a dramatic increase in the number of SRs in recent decade; however no significant improvement was seen in search strategies during the time. The mean score of search strategies among SRs was 4.63 (out of 9). The highest score was 7 (21.2%) and more than 53% of SRs obtained a score between 5 and 7. In 74.4% of SRs, 3–8 databases were searched which 40.4% were considered 4 databases. About 55% of SRs reported search keywords and using Boolean operators that has increased in recent years. Near to 98% of studies were not conducted in duplicate. In near to 75% of studies hand-searching was performed but instead, in 72.3% grey literature was not searched and in 65.9% the authors of studies were not contacted for further relevant information. Fortunately, in about 60% of studies the literature search was performed without language restriction. Conclusions: Lack of duplicate literature searching in systematic reviews was notable. Although most of SRs were searched more than four databases but it seems that in most of them the major databases were not searched and instead, local or specialized databases were considered. Low tendency to grey literature searching was remarkable. It seems that the quality of reporting search strategies in the field of DSS is poor or moderate.

medicine, general & internal
Evaluating Search Systems through Field Codes and Classifications: The Case of the First Published Swedish Nursing Study

Christopher Holmberg

DOI: https://doi.org/10.48550/arXiv.2311.15349

2023-11-26

Digital Libraries

Abstract:The aim of the study was to assess and compare established search systems and approaches by using the search goal of identifying the first (as in oldest) nursing-related document, with reference to the first Swedish-affiliated document, in Scopus, Web of Science, and PubMed (Medline). In doing so, the objective was to provide concrete examples and illustrate how search systems and field code searches versus classification-based searches differ. The Scopus database, Web of Science database aggregator, and the PubMed (Medline) search engine were used for evaluations. Two different search strategies were compared in each database: one guided by field codes (i.e., text-based), and one guided by pre-existing categorizations within database infrastructures. Findings highlight several factors that are important to consider when formulating and executing a search strategy. The findings illustrate important aspects of search systems and search approaches, namely the publication year, delimitations related to a country or region, and features related to a subject or research area. The findings also highlight the importance of prioritizing between different ideals in retrieval. For example, is it more important to reach the oldest records (publication year), to identify as many records as possible (exhaustiveness), or to make sure they are as subject-relevant as possible (e.g., in terms of authors belonging to the same field)? Secondly, researchers using bibliometric methods to analyze research literature should be more transparent when reporting searches, as different search systems and search approaches yield varied numbers and quality of records. While most bibliometric researchers strive to combine different approaches, this is not always made clear in reporting.
Finding and Recommending Scholarly Articles

Michael J. Kurtz,Edwin A. Henneken

DOI: https://doi.org/10.7551/mitpress/9445.001.0001

2012-09-06

Abstract:The rate at which scholarly literature is being produced has been increasing at approximately 3.5 percent per year for decades. This means that during a typical 40 year career the amount of new literature produced each year increases by a factor of four. The methods scholars use to discover relevant literature must change. Just like everybody else involved in information discovery, scholars are confronted with information overload. Two decades ago, this discovery process essentially consisted of paging through abstract books, talking to colleagues and librarians, and browsing journals. A time-consuming process, which could even be longer if material had to be shipped from elsewhere. Now much of this discovery process is mediated by online scholarly information systems. All these systems are relatively new, and all are still changing. They all share a common goal: to provide their users with access to the literature relevant to their specific needs. To achieve this each system responds to actions by the user by displaying articles which the system judges relevant to the user's current needs. Recently search systems which use particularly sophisticated methodologies to recommend a few specific papers to the user have been called "recommender systems". These methods are in line with the current use of the term "recommender system" in computer science. We do not adopt this definition, rather we view systems like these as components in a larger whole, which is presented by the scholarly information systems themselves. In what follows we view the recommender system as an aspect of the entire information system; one which combines the massive memory capacities of the machine with the cognitive abilities of the human user to achieve a human-machine synergy.

Information Retrieval,Instrumentation and Methods for Astrophysics,Digital Libraries,Physics and Society
Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases

Michael Gusenbauer

DOI: https://doi.org/10.1007/s11192-018-2958-5

IF: 3.801

2018-11-10

Scientometrics

Abstract:Information on the size of academic search engines and bibliographic databases (ASEBDs) is often outdated or entirely unavailable. Hence, it is difficult to assess the scope of specific databases, such as Google Scholar. While scientometric studies have estimated ASEBD sizes before, the methods employed were able to compare only a few databases. Consequently, there is no up-to-date comparative information on the sizes of popular ASEBDs. This study aims to fill this blind spot by providing a comparative picture of 12 of the most commonly used ASEBDs. In doing so, we build on and refine previous scientometric research by counting query hit data as an indicator of the number of accessible records. Iterative query optimization makes it possible to identify a maximum number of hits for most ASEBDs. The results were validated in terms of their capacity to assess database size by comparing them with official information on database sizes or previous scientometric studies. The queries used here are replicable, so size information can be updated quickly. The findings provide first-time size estimates of ProQuest and EbscoHost and indicate that Google Scholar’s size might have been underestimated so far by more than 50%. By our estimation Google Scholar, with 389 million records, is currently the most comprehensive academic search engine.

information science & library science,computer science, interdisciplinary applications
Identifying Relevant Studies in Software Engineering

He Zhang,Muhammad All Babar,Paolo Tell

DOI: https://doi.org/10.1016/j.infsof.2010.12.010

IF: 3.9

2011-01-01

Information and Software Technology

Abstract:Context: Systematic literature review (SLR) has become an important research methodology in software engineering since the introduction of evidence-based software engineering (EBSE) in 2004. One critical step in applying this methodology is to design and execute appropriate and effective search strategy. This is a time-consuming and error-prone step, which needs to be carefully planned and implemented. There is an apparent need for a systematic approach to designing, executing, and evaluating a suitable search strategy for optimally retrieving the target literature from digital libraries.Objective: The main objective of the research reported in this paper is to improve the search step of undertaking SLRs in software engineering (SE) by devising and evaluating systematic and practical approaches to identifying relevant studies in SE.Method: We have systematically selected and analytically studied a large number of papers (SLRs) to understand the state-of-the-practice of search strategies in EBSE. Having identified the limitations of the current ad-hoc nature of search strategies used by SE researchers for SLRs, we have devised a systematic and evidence-based approach to developing and executing optimal search strategies in SLRs. The proposed approach incorporates the concept of 'quasi-gold standard' (QGS), which consists of collection of known studies, and corresponding 'quasi-sensitivity' into the search process for evaluating search performance.Results: We conducted two participant-observer case studies to demonstrate and evaluate the adoption of the proposed QGS-based systematic search approach in support of SLRs in SE research.Conclusion: We report their findings based on the case studies that the approach is able to improve the rigor of search process in an SLR, as well as it can serve as a supplement to the guidelines for SLRs in EBSE. We plan to further evaluate the proposed approach using a series of case studies on varying research topics in SE. (C) 2011 Elsevier B.V. All rights reserved.
A Technical Review of the ISPOR Presentations Database Identified Issues in the Search Interface and Areas for Future Development

Chris Cooper,Anna Brown,Rachel Court,Ute Schauberger

DOI: https://doi.org/10.1017/S0266462322000137

2022-03-08

Abstract:Objective: To undertake a technical review of the search interface of the ISPOR Presentations Database. By technical review, we mean an evaluation of the technical aspects of the search interface and functionality, which a user must navigate to complete a search. Methods: A validated checklist (Bethel and Rogers, 2014, Health Info Libr J, 31, 43-53) was used to identify where the interface performed well, where the interface was adequate, where the interface performed poorly, where functionality available in core biomedical bibliographic databases does not exist in the ISPOR database, and to establish a list of any issues arising during the review. Two researchers independently undertook the technical review in October 2021. Results: The ISPOR database scored 35 of a possible 165 (27/111 essential criteria and 8/54 desirable criteria). Two issues arising were identified, both of which will cause searchers to miss potentially eligible abstracts: (i) that search terms, which include * or ? as truncation or wildcard symbols should not be capitalized (e.g., cost* not Cost*; organi?ation not Organi?ation) and (ii) that quotation marks should be straight sided in phrase searching (e.g., "cost analyses" not "cost analyses"). Conclusions: The ISPOR database is a promising and free database to identify abstracts/posters presented at ISPOR. We summarize two key issues arising, and we set out proposed changes to the search interface, including: adding the ability to export abstracts to a bibliographic tool, exporting search strategies, adding a researcher account, and updating the help guide. All suggestions will further improve this helpful database.
On the performance of hybrid search strategies for systematic literature reviews in software engineering

Erica Mourão,João Felipe Pimentel,Leonardo Murta,Marcos Kalinowski,Emilia Mendes,Claes Wohlin

DOI: https://doi.org/10.1016/j.infsof.2020.106294

IF: 3.9

2020-07-01

Information and Software Technology

Abstract:<h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Context</h3><p>When conducting a Systematic Literature Review (SLR), researchers usually face the challenge of designing a search strategy that appropriately balances result quality and review effort. Using digital library (or database) searches or snowballing alone may not be enough to achieve high-quality results. On the other hand, using both digital library searches and snowballing together may increase the overall review effort.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Objective</h3><p>The goal of this research is to propose and evaluate hybrid search strategies that selectively combine database searches with snowballing.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Method</h3><p>We propose four hybrid search strategies combining database searches in digital libraries with iterative, parallel, or sequential backward and forward snowballing. We simulated the strategies over three existing SLRs in SE that adopted both database searches and snowballing. We compared the outcome of digital library searches, snowballing, and hybrid strategies using precision, recall, and F-measure to investigate the performance of each strategy.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Results</h3><p>Our results show that, for the analyzed SLRs, combining database searches from the Scopus digital library with parallel or sequential snowballing achieved the most appropriate balance of precision and recall.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Conclusion</h3><p>We put forward that, depending on the goals of the SLR and the available resources, using a hybrid search strategy involving a representative digital library and parallel or sequential snowballing tends to represent an appropriate alternative to be used when searching for evidence in SLRs.</p>

computer science, information systems, software engineering
Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers

Neil R Smalheiser,Can Lin,Lifeng Jia,Yu Jiang,Aaron M Cohen,Clement Yu,John M Davis,Clive E Adams,Marian S McDonagh,Weiyi Meng

DOI: https://doi.org/10.1186/2047-2501-2-1

2014-01-10

Health Information Science and Systems

Abstract:BackgroundIndividuals and groups who write systematic reviews and meta-analyses in evidence-based medicine regularly carry out literature searches across multiple search engines linked to different bibliographic databases, and thus have an urgent need for a suitable metasearch engine to save time spent on repeated searches and to remove duplicate publications from initial consideration. Unlike general users who generally carry out searches to find a few highly relevant (or highly recent) articles, systematic reviewers seek to obtain a comprehensive set of articles on a given topic, satisfying specific criteria. This creates special requirements and challenges for metasearch engine design and implementation.MethodsWe created a federated search tool that is connected to five databases: PubMed, EMBASE, CINAHL, PsycINFO, and the Cochrane Central Register of Controlled Trials. Retrieved bibliographic records were shown online; optionally, results could be de-duplicated and exported in both BibTex and XML format.ResultsThe query interface was extensively modified in response to feedback from users within our team. Besides a general search track and one focused on human-related articles, we also added search tracks optimized to identify case reports and systematic reviews. Although users could modify preset search options, they were rarely if ever altered in practice. Up to several thousand retrieved records could be exported within a few minutes. De-duplication of records returned from multiple databases was carried out in a prioritized fashion that favored retaining citations returned from PubMed.ConclusionsSystematic reviewers are used to formulating complex queries using strategies and search tags that are specific for individual databases. Metta offers a different approach that may save substantial time but which requires modification of current search strategies and better indexing of randomized controlled trial articles. We envision Metta as one piece of a multi-tool pipeline that will assist systematic reviewers in retrieving, filtering and assessing publications. As such, Metta may find wide utility for anyone who is carrying out a comprehensive search of the biomedical literature.

Effects of prostaglandin B2 on vascular tone and reactivity.

Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Irreproducibility in Searches of Scientific Literature: A Comparative Analysis.

Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Beyond Google Scholar, Scopus, and Web of Science: An evaluation of the backward and forward citation coverage of 59 databases' citation indices

The optimal approach for retrieving systematic reviews was achieved when searching MEDLINE and Epistemonikos in addition to reference checking: a methodological validation study

Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses

Where you search determines what you find: the effects of bibliographic databases on systematic reviews

Systematically Searching Empirical Literature in the Social Sciences: Results from Two Meta-Analyses Within the Domain of Education

The contribution of databases to the results of systematic reviews: a cross-sectional study

Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies

The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

A systematic approach to searching: an efficient and complete method to develop literature searches

158: ASSESSING THE QUALITY OF SEARCH STRATEGIES IN DECISION SUPPORT SYSTEM (DSS) SYSTEMATIC REVIEWS

Evaluating Search Systems through Field Codes and Classifications: The Case of the First Published Swedish Nursing Study

Finding and Recommending Scholarly Articles

Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases

Identifying Relevant Studies in Software Engineering

A Technical Review of the ISPOR Presentations Database Identified Issues in the Search Interface and Areas for Future Development

On the performance of hybrid search strategies for systematic literature reviews in software engineering

Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers