Endoscopic aspects of the gastric mucosa in relation to its function. I. In the gastritides.

S. Stempien,J. E. Oritt,N. W. Karr

DOI: https://doi.org/10.1016/S0016-5085(53)80086-7

IF: 29.4

Gastroenterology

Abstract:

What problem does this paper attempt to address?

[Number Needed to Be Screened in a Study: a Novel Measure for Disease Screening Effect].

Feng Tong,Kun Chen,Han-qing He

DOI: https://doi.org/10.3760/j.issn:0254-6450.2006.08.021

2006-01-01

Abstract:OBJECTIVETo introduce the concept, methods for calculation and application of "number needed to be screened" (NNBS) in epidemiologic studies.METHODSThe concept of "number needed to treat" (NNT) was extended for disease screening strategies. For the purpose of illustration, the values of number needed to invite for screening (NNI) and number needed to be screened (NNBS) were calculated on the basis of the results from two randomized controlled screening trials--Nottingham randomized controlled trial of faecal-occult-blood screening for colorectal cancer and Swedish mammographic screening trial for breast cancer in two counties.RESULTSIn order to prevent one death from the colorectal cancer among local people aged from 45 to 74 during the 14 years of follow-up, the NNI and NNBS for faecal-occult-blood screening program were 1220 and 665, respectively. In addition, in order to prevent one death from breast cancer among local women aged 40-74 during 8 years of follow-up, the NNI and NNBS for mammographic screening program were 1961 and 1494, respectively.CONCLUSIONCompared with the traditional indices, NNBS can evaluate the overall effectiveness of a screening program in an intuitively understandable manner so as to facilitate the communication among medical researchers, health workers, health policy makers and the public.
Irreproducibility in Searches of Scientific Literature: A Comparative Analysis.

Gabor Pozsgai,Gabor L. Lovei,Liette Vasseur,Geoff Gurr,Peter Batary,Janos Korponai,Nick A. Littlewood,Jian Liu,Arnold Mora,John Obrycki,Olivia Reynolds,Jenni A. Stockan,Heather VanVolkenburg,Jie Zhang,Wenwu Zhou,Minsheng You

DOI: https://doi.org/10.1002/ece3.8154

2020-01-01

Abstract:Abstract Repeatability is the cornerstone of science, and it is particularly important for systematic reviews. However, little is known on how researchers’ choice of database, and search platform influence the repeatability of systematic reviews. Here, we aim to unveil how the computer environment and the location where the search was initiated from influence hit results. We present a comparative analysis of time‐synchronized searches at different institutional locations in the world and evaluate the consistency of hits obtained within each of the search terms using different search platforms. We revealed a large variation among search platforms and showed that PubMed and Scopus returned consistent results to identical search strings from different locations. Google Scholar and Web of Science's Core Collection varied substantially both in the number of returned hits and in the list of individual articles depending on the search location and computing environment. Inconsistency in Web of Science results has most likely emerged from the different licensing packages at different institutions. To maintain scientific integrity and consistency, especially in systematic reviews, action is needed from both the scientific community and scientific search platforms to increase search consistency. Researchers are encouraged to report the search location and the databases used for systematic reviews, and database providers should make search algorithms transparent and revise access rules to titles behind paywalls. Additional options for increasing the repeatability and transparency of systematic reviews are storing both search metadata and hit results in open repositories and using Application Programming Interfaces (APIs) to retrieve standardized, machine‐readable search metadata.
Increasing the efficiency of study selection for systematic reviews using prioritization tools and a single-screening approach

Siw Waffenschmidt,Wiebke Sieben,Thomas Jakubeit,Marco Knelangen,Inga Overesch,Stefanie Bühn,Dawid Pieper,Nicole Skoetz,Elke Hausner

DOI: https://doi.org/10.1186/s13643-023-02334-x

2023-09-14

Systematic Reviews

Abstract:Abstract Background Systematic literature screening is a key component in systematic reviews. However, this approach is resource intensive as generally two persons independently of each other (double screening) screen a vast number of search results. To develop approaches for increasing efficiency, we tested the use of text mining to prioritize search results as well as the involvement of only one person (single screening) in the study selection process. Method Our study is based on health technology assessments (HTAs) of drug and non-drug interventions. Using a sample size calculation, we consecutively included 11 searches resulting in 33 study selection processes. Of the three screeners for each search, two used screening tools with prioritization (Rayyan, EPPI Reviewer) and one a tool without prioritization. For each prioritization tool, we investigated the proportion of citations classified as relevant at three cut-offs or STOP criteria (after screening 25%, 50% and 75% of the citation set). For each STOP criterion, we measured sensitivity (number of correctly identified relevant studies divided by the total number of relevant studies in the study pool). In addition, we determined the number of relevant studies identified per single screening round and investigated whether missed studies were relevant to the HTA conclusion. Results Overall, EPPI Reviewer performed better than Rayyan and identified the vast majority (88%, Rayyan 66%) of relevant citations after screening half of the citation set. As long as additional information sources were screened, it was sufficient to apply a single-screening approach to identify all studies relevant to the HTA conclusion. Although many relevant publications ( n = 63) and studies ( n = 29) were incorrectly excluded, ultimately only 5 studies could not be identified at all in 2 of the 11 searches (1x 1 study, 1x 4 studies). However, their omission did not change the overall conclusion in any HTA. Conclusions EPPI Reviewer helped to identify relevant citations earlier in the screening process than Rayyan. Single screening would have been sufficient to identify all studies relevant to the HTA conclusion. However, this requires screening of further information sources. It also needs to be considered that the credibility of an HTA may be questioned if studies are missing, even if they are not relevant to the HTA conclusion.

medicine, general & internal
Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Michael Gusenbauer,Neal R. Haddaway

DOI: https://doi.org/10.1002/jrsm.1378

2020-01-28

Research Synthesis Methods

Abstract:Rigorous evidence identification is essential for systematic reviews and meta‐analyses (evidence syntheses), because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments.This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed and Web of Science. A novel, query‐based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analysed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system.We call for database owners to recognise the requirements of evidence synthesis, and for academic journals to re‐assess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

multidisciplinary sciences,mathematical & computational biology
UN Development Programme and non-communicable diseases

P. Lal,A. Jacob,A. Buragohain

DOI: https://doi.org/10.1016/S0140-6736(13)61628-1

IF: 202.731

2013-07-27

The Lancet

Abstract:
The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

Neal Robert Haddaway,Alexandra Mary Collins,Deborah Coughlin,Stuart Kirk

DOI: https://doi.org/10.1371/journal.pone.0138237

IF: 3.7

2015-09-17

PLoS ONE

Abstract:Google Scholar (GS), a commonly used web-based academic search engine, catalogues between 2 and 100 million records of both academic and grey literature (articles not formally published by commercial academic publishers). Google Scholar collates results from across the internet and is free to use. As a result it has received considerable attention as a method for searching for literature, particularly in searches for grey literature, as required by systematic reviews. The reliance on GS as a standalone resource has been greatly debated, however, and its efficacy in grey literature searching has not yet been investigated. Using systematic review case studies from environmental science, we investigated the utility of GS in systematic reviews and in searches for grey literature. Our findings show that GS results contain moderate amounts of grey literature, with the majority found on average at page 80. We also found that, when searched for specifically, the majority of literature identified using Web of Science was also found using GS. However, our findings showed moderate/poor overlap in results when similar search strings were used in Web of Science and GS (10-67%), and that GS missed some important literature in five of six case studies. Furthermore, a general GS search failed to find any grey literature from a case study that involved manual searching of organisations' websites. If used in systematic reviews for grey literature, we recommend that searches of article titles focus on the first 200 to 300 results. We conclude that whilst Google Scholar can find much grey literature and specific, known studies, it should not be used alone for systematic review searches. Rather, it forms a powerful addition to other traditional search methods. In addition, we advocate the use of tools to transparently document and catalogue GS search results to maintain high levels of transparency and the ability to be updated, critical to systematic reviews.

multidisciplinary sciences
Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study

Shuai Wang,Harrisen Scells,Ahmed Mourad,Guido Zuccon

DOI: https://doi.org/10.48550/arXiv.2112.04090

2021-12-08

Abstract:Screening or assessing studies is critical to the quality and outcomes of a systematic review. Typically, a Boolean query retrieves the set of studies to screen. As the set of studies retrieved is unordered, screening all retrieved studies is usually required for high-quality systematic reviews. Screening prioritisation, or in other words, ranking the set of studies, enables downstream activities of a systematic review to begin in parallel. We investigate a method that exploits seed studies -- potentially relevant studies used to seed the query formulation process -- for screening prioritisation. Our investigation aims to reproduce this method to determine if it is generalisable on recently published datasets and determine the impact of using multiple seed studies on <a class="link-external link-http" href="http://effectiveness.We" rel="external noopener nofollow">this http URL</a> show that while we could reproduce the original methods, we could not replicate their results exactly. However, we believe this is due to minor differences in document pre-processing, not deficiencies with the original methodology. Our results also indicate that our reproduced screening prioritisation method, (1) is generalisable across datasets of similar and different topicality compared to the original implementation, (2) that when using multiple seed studies, the effectiveness of the method increases using our techniques to enable this, (3) and that the use of multiple seed studies produces more stable rankings compared to single seed studies. Finally, we make our implementation and results publicly available at the following URL: <a class="link-external link-https" href="https://github.com/ielab/sdr" rel="external noopener nofollow">this https URL</a>

Information Retrieval
The optimal approach for retrieving systematic reviews was achieved when searching MEDLINE and Epistemonikos in addition to reference checking: a methodological validation study

Lena Heinen,Käthe Goossen,Carole Lunny,Julian Hirt,Livia Puljak,Dawid Pieper

DOI: https://doi.org/10.1186/s12874-024-02384-2

2024-11-10

BMC Medical Research Methodology

Abstract:Systematic reviews (SRs) are used to inform clinical practice guidelines and healthcare decision making by synthesising the results of primary studies. Efficiently retrieving as many relevant SRs as possible is challenging with a minimum number of databases, as there is currently no guidance on how to do this optimally. In a previous study, we determined which individual databases contain the most SRs, and which combination of databases retrieved the most SRs. In this study, we aimed to validate those previous results by using a different, larger, and more recent set of SRs.

health care sciences & services
Efficient searching for NICE Public Health Guidelines: would using fewer sources still find the evidence?

Paul Levay,Andrea Heath,Daniel Tuvey

DOI: https://doi.org/10.1002/jrsm.1577

2022-06-04

Research Synthesis Methods

Abstract:Background Systematic searches are integral to identifying the evidence that is used in National Institute for Health and Care Excellence (NICE) public health guidelines (PHGs). This study analyses the sources, including bibliographic databases and other techniques, required for PHGs. Aims To analyse the sources used to identify the publications included in NICE PHGs; and to assess whether fewer sources could have been searched to retrieve these publications. Methods Data showing how the included publications had been identified was collated using search summary tables. Three scenarios were created to test various combinations of sources to determine whether fewer sources could have been used. The sample included 29 evidence reviews, compiled using 13 searches, to support 10 PHG topics. Results Across the PHGs, 23 databases and six other techniques retrieved included publications. A mean reduction in total results of 6.5% could have been made if the minimum set of sources plus Cochrane Library, Embase and MEDLINE were searched. On average, Cochrane Library, Embase and MEDLINE contributed 76.8% of the included publications, with other databases adding 11% and other techniques 12.2%. Discussion None of the searches had a minimum set that was comprised entirely of databases. There was not a core set of sources for PHGs. Conclusions A range of databases and techniques, covering a multi‐disciplinary evidence base, was required to identify all included publications. It would be possible to reduce the number of sources searched and make some gains in productivity. It is important to create a tailored set of sources to do an efficient search. This article is protected by copyright. All rights reserved.

multidisciplinary sciences,mathematical & computational biology
How much searching is enough? Comprehensive versus optimal retrieval for technology assessments

Andrew Booth

DOI: https://doi.org/10.1017/s0266462310000966

2010-10-01

International Journal of Technology Assessment in Health Care

Abstract:Objectives: The aim of this study is to review briefly different methods for determining the optimal retrieval of studies for inclusion in a health technology assessment (HTA) report. Methods: This study reviews the methodology literature related to specific methods for evaluating yield from literature searching strategies and for deciding whether to continue or desist in the searching process. Results: Eight different methods were identified. These include using the Capture–recapture technique; obtaining Feedback from the commissioner of the HTA report; seeking the Disconfirming case; undertaking comparison against a known Gold standard; evaluating retrieval of Known items; recognizing the Law of diminishing returns, specifying a priori Stopping rules, and identifying a point of Theoretical saturation. Conclusions: While this study identified a variety of possible methods, there has been very little formal evaluation of the specific strengths and weaknesses of the different techniques. The author proposes an evaluation agenda drawing on an examination of existing data together with exploration of the specific impact of missing relevant studies.

public, environmental & occupational health,health care sciences & services,medical informatics
Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Wichor M. Bramer,Melissa L. Rethlefsen,Jos Kleijnen,Oscar H. Franco

DOI: https://doi.org/10.1186/s13643-017-0644-y

2017-12-01

Systematic Reviews

Abstract:BackgroundWithin systematic reviews, when searching for relevant references, it is advisable to use multiple databases. However, searching databases is laborious and time-consuming, as syntax of search strategies are database specific. We aimed to determine the optimal combination of databases needed to conduct efficient searches in systematic reviews and whether the current practice in published reviews is appropriate. While previous studies determined the coverage of databases, we analyzed the actual retrieval from the original searches for systematic reviews.MethodsSince May 2013, the first author prospectively recorded results from systematic review searches that he performed at his institution. PubMed was used to identify systematic reviews published using our search strategy results. For each published systematic review, we extracted the references of the included studies. Using the prospectively recorded results and the studies included in the publications, we calculated recall, precision, and number needed to read for single databases and databases in combination. We assessed the frequency at which databases and combinations would achieve varying levels of recall (i.e., 95%). For a sample of 200 recently published systematic reviews, we calculated how many had used enough databases to ensure 95% recall.ResultsA total of 58 published systematic reviews were included, totaling 1746 relevant references identified by our database searches, while 84 included references had been retrieved by other search methods. Sixteen percent of the included references (291 articles) were only found in a single database; Embase produced the most unique references (n = 132). The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar performed best, achieving an overall recall of 98.3 and 100% recall in 72% of systematic reviews. We estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references as many fail to search important databases. Other specialized databases, such as CINAHL or PsycINFO, add unique references to some reviews where the topic of the review is related to the focus of the database.ConclusionsOptimal searches in systematic reviews should search at least Embase, MEDLINE, Web of Science, and Google Scholar as a minimum requirement to guarantee adequate and efficient coverage.

medicine, general & internal
An Empirical Assessment of a Systematic Search Process for Systematic Reviews

He Zhang,Muhammad Ali Babar,Xu Bai,Juan Li,Liguo Huang

DOI: https://doi.org/10.1049/ic.2011.0007

2011-01-01

Abstract:Background: Systematic Literature Reviews (SLRs) have been gaining significant attention from Software Engineering (SE) researchers. Several researchers are also working on improving SLR methodology for SE. Objective: The study reported in this paper aims to validate the QGS-based search process for SLR, i.e. whether a more effective (sensitive) and/or productive (precise) search can be achieved by following the QGS-based process. Method: We used a dual-case study, in which each case includes two observations of SE literature search for the same SLR but using and not using the QGS-based approach. The overall sensitivity and precision were calculated for each observation that implemented different search design. Results: The use of QGS-based search process resulted in higher sensitivity and precision in this dual-case study. Conclusions: A systematic search process can help capture more relevant studies as well as save researchers' time spent on literature search activities. Our observations also support that an integrated search strategy is recommended for SLRs in SE to avoid the possible limitations of applying a single (manual or automated) search method.
Abstract 1265: Comparison of mortality- vs. stage-based endpoints in randomized trials of cancer screening: A systematic review with implications for multi-cancer screening trials

Xiaoshuang Feng,Hana Zahed,Justina Onwuka,Mattias Johansson,Ruth Etzioni,Matthew E. Callister,Hilary A. Robbins

DOI: https://doi.org/10.1158/1538-7445.am2024-1265

IF: 11.2

2024-03-23

Cancer Research

Abstract:Background: Randomized controlled trials (RCTs) assessing the endpoint of late-stage cancer incidence, rather than the gold-standard of cancer-specific mortality, are proposed for evaluation of multi-cancer early detection tests. Methods: We conducted a systematic review to assess whether stage-based endpoints can provide a valid alternative for mortality-based endpoints (registration: PROSPERO CRD42023411132). We searched PubMed to identify RCTs of cancer screening interventions published in English before January 20, 2023. Inclusion required that RCTs report, by study arm, numbers of cancer-specific deaths and cancer cases distinguishable between stages I-II vs. III-IV. Data extraction was done by 2 separate reviewers and included the numbers of cancer cases, deaths, and participants by study arm. For each trial, we calculated the percentage reduction in the cumulative (a) late-stage cancer incidence and (b) cancer-specific mortality between the intervention and control arms. We compared these two endpoints based on Pearson correlation coefficients, fixed-effects meta-analysis, and binary statistical significance. Results: We analyzed 41 trials ranging in size from 2,500 to 409,000 participants. Correlation between the reduction in cancer-specific mortality vs. stage III-IV cancer was strong for lung (Pearson's ρ=0.92, n=12), breast (ρ=0.70, n=6), and 'other' cancers (ρ=0.69, n=8) but weak for colorectal (ρ=0.39, n=11) and prostate cancers (ρ<0, n=4). The overall correlation by fixed-effects meta-analysis across cancer types was 0.71 (95%CI=0.47-0.86) (I2=0.55). In larger trials, there was a smaller deviation between the observed reduction in cancer-specific mortality and that predicted, by cancer type, based on the trial's reduction in stage III-IV cancer (p=0.02). Of 12 trials demonstrating a statistically significant reduction in stage III-IV cancer incidence (p<0.05), 5 also showed reduced cancer-specific mortality (42%). Conclusions: In screening RCTs, reductions in late-stage cancer vs. cancer-specific mortality are correlated, but with important differences by cancer type. Screening interventions that reduce late-stage cancer incidence do not necessarily reduce cancer-specific mortality. Citation Format: Xiaoshuang Feng, Hana Zahed, Justina Onwuka, Mattias Johansson, Ruth Etzioni, Matthew E. Callister, Hilary A. Robbins. Comparison of mortality- vs. stage-based endpoints in randomized trials of cancer screening: A systematic review with implications for multi-cancer screening trials [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 1265.

oncology
158: ASSESSING THE QUALITY OF SEARCH STRATEGIES IN DECISION SUPPORT SYSTEM (DSS) SYSTEMATIC REVIEWS

Zahra Fathifar,Hossein Saeedi Hasankandi,Fatemeh Sadeghi-Ghyassi

DOI: https://doi.org/10.1136/bmjopen-2016-015415.158

IF: 3.006

2017-02-01

BMJ Open

Abstract:Background and aims: Search strategies play a major role in developing a systematic review. A comprehensive literature search dramatically reduces publication bias. The higher sensitive search leads to a more well-prepared systematic review. The aim of this study is evaluating the quality of search strategies in decision support system (DSS) systematic reviews. Methods: A comprehensive literature search using MEDLINE (via Ovid), Embase, Pubmed and Cochrane Library was conducted until September August 2016. The search was limited to systematic reviews and with no language restriction. Meta-reviews were excluded. Only results of searching for “decision support systems” or “decision support system” keywords and limited to title of documents were included in this study. Two independent authors screened the results. Disagreement was resolved by discussion. The search strategies of the included studies were assessed with a nine-question checklist prepared by Faggion et al 1 . Data were extracted into Excel. Results: A total of 47 systematic reviews (SRs) were included. There was a dramatic increase in the number of SRs in recent decade; however no significant improvement was seen in search strategies during the time. The mean score of search strategies among SRs was 4.63 (out of 9). The highest score was 7 (21.2%) and more than 53% of SRs obtained a score between 5 and 7. In 74.4% of SRs, 3–8 databases were searched which 40.4% were considered 4 databases. About 55% of SRs reported search keywords and using Boolean operators that has increased in recent years. Near to 98% of studies were not conducted in duplicate. In near to 75% of studies hand-searching was performed but instead, in 72.3% grey literature was not searched and in 65.9% the authors of studies were not contacted for further relevant information. Fortunately, in about 60% of studies the literature search was performed without language restriction. Conclusions: Lack of duplicate literature searching in systematic reviews was notable. Although most of SRs were searched more than four databases but it seems that in most of them the major databases were not searched and instead, local or specialized databases were considered. Low tendency to grey literature searching was remarkable. It seems that the quality of reporting search strategies in the field of DSS is poor or moderate.

medicine, general & internal
An evidence-based practice guideline for the peer review of electronic search strategies

Margaret Sampson,Jessie McGowan,Elise Cogo,Jeremy Grimshaw,David Moher,Carol Lefebvre

DOI: https://doi.org/10.1016/j.jclinepi.2008.10.012

IF: 7.407

2009-09-01

Journal of Clinical Epidemiology

Abstract:OBJECTIVE: Complex and highly sensitive electronic literature search strategies are required for systematic reviews; however, no guidelines exist for their peer review. Poor searches may fail to identify existing evidence because of inadequate recall (sensitivity) or increase the resource requirements of reviews as a result of inadequate precision. Our objective was to create an annotated checklist for electronic search strategy peer review.STUDY DESIGN: A systematic review of the library and information retrieval literature for important elements in electronic search strategies was conducted, along with a survey of individuals experienced in systematic review searching.RESULTS: Six elements with a strong consensus as to their importance in peer review were accurate translation of the research question into search concepts, correct choice of Boolean operators and of line numbers, adequate translation of the search strategy for each database, inclusion of relevant subject headings, and absence of spelling errors. Seven additional elements had partial support and are included in this guideline.CONCLUSION: This evidence-based guideline facilitates the improvement of search quality through peer review, and thus the improvement in quality of systematic reviews. It is relevant for librarians/information specialists, journal editors, developers of knowledge translation tools, research organizations, and funding bodies.

public, environmental & occupational health,health care sciences & services
Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies

Claes Wohlin,Marcos Kalinowski,Katia Romero Felizardo,Emilia Mendes

DOI: https://doi.org/10.1016/j.infsof.2022.106908

2023-07-06

Abstract:Background: A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which has later been complemented with snowball searches. Our conjecture is that we can perform even better searches if combining the two search approaches, referred to as a hybrid search strategy. Objective: Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared some alternative hybrid search strategies to assess whether it was possible to identify more cost-efficient ways of searching for relevant primary studies. Method: To compare and evaluate the hybrid search strategy, we replicated an SLR on industry-academia collaboration in software engineering. The SLR used a more traditional approach to searching for relevant articles for an SLR, while the replication was conducted using a hybrid search strategy. Results: In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30 percent more primary studies and even more when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when conducting systematic literature studies. Conclusions: The hybrid search strategy is a strong contender for being used when conducting systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.

Software Engineering
Novel citation-based search method for scientific literature: application to meta-analyses

A. Cecile J W Janssens,M. Gwinn

DOI: https://doi.org/10.1186/s12874-015-0077-z

2015-10-13

BMC Medical Research Methodology

Abstract:BackgroundFinding eligible studies for meta-analysis and systematic reviews relies on keyword-based searching as the gold standard, despite its inefficiency. Searching based on direct citations is not sufficiently comprehensive. We propose a novel strategy that ranks articles on their degree of co-citation with one or more “known” articles before reviewing their eligibility.MethodIn two independent studies, we aimed to reproduce the results of literature searches for sets of published meta-analyses (n = 10 and n = 42). For each meta-analysis, we extracted co-citations for the randomly selected ‘known’ articles from the Web of Science database, counted their frequencies and screened all articles with a score above a selection threshold. In the second study, we extended the method by retrieving direct citations for all selected articles.ResultsIn the first study, we retrieved 82 % of the studies included in the meta-analyses while screening only 11 % as many articles as were screened for the original publications. Articles that we missed were published in non-English languages, published before 1975, published very recently, or available only as conference abstracts. In the second study, we retrieved 79 % of included studies while screening half the original number of articles.ConclusionsCitation searching appears to be an efficient and reasonably accurate method for finding articles similar to one or more articles of interest for meta-analysis and reviews.

health care sciences & services
The contribution of databases to the results of systematic reviews: a cross-sectional study

Lisa Hartling,Robin Featherstone,Megan Nuspl,Kassi Shave,Donna M. Dryden,Ben Vandermeer

DOI: https://doi.org/10.1186/s12874-016-0232-1

2016-09-26

BMC Medical Research Methodology

Abstract:BackgroundOne of the best sources for high quality information about healthcare interventions is a systematic review. A well-conducted systematic review includes a comprehensive literature search. There is limited empiric evidence to guide the extent of searching, in particular the number of electronic databases that should be searched. We conducted a cross-sectional quantitative analysis to examine the potential impact of selective database searching on results of meta-analyses.MethodsOur sample included systematic reviews (SRs) with at least one meta-analysis from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of relevant studies indexed in each of 10 databases; and 2) changes in results and statistical significance of primary meta-analysis for studies identified in Medline only and in Medline plus each of the other databases.ResultsDue to variation across topics, we present results by group (ARI n = 57, ID n = 38, DPLP n = 34). For ARI, identification of relevant studies was highest for Medline (85 %) and Embase (80 %). Restricting meta-analyses to trials that appeared in Medline + Embase yielded fewest changes in statistical significance: 53/55 meta-analyses showed no change. Point estimates changed in 12 cases; in 7 the change was less than 20 %. For ID, yield was highest for Medline (92 %), Embase (81 %), and BIOSIS (67 %). Restricting meta-analyses to trials that appeared in Medline + BIOSIS yielded fewest changes with 1 meta-analysis changing in statistical significance. Point estimates changed in 8 of 31 meta-analyses; change less than 20 % in all cases. For DPLP, identification of relevant studies was highest for Medline (75 %) and Embase (62 %). Restricting meta-analyses to trials that appeared in Medline + PsycINFO resulted in only one change in significance. Point estimates changed for 13 of 33 meta-analyses; less than 20 % in 9 cases.ConclusionsMajority of relevant studies can be found within a limited number of databases. Results of meta-analyses based on the majority of studies did not differ in most cases. There were very few cases of changes in statistical significance. Effect estimates changed in a minority of meta-analyses but in most the change was small. Results did not change in a systematic manner (i.e., regularly over- or underestimating treatment effects), suggesting that selective searching may not introduce bias in terms of effect estimates.

health care sciences & services
Prioritising references for systematic reviews with RobotAnalyst: A user study

Piotr Przybyła,Austin J. Brockmeier,Georgios Kontonatsios,Marie‐Annick Le Pogam,John McNaught,Erik Elm,Kay Nolan,Sophia Ananiadou

DOI: https://doi.org/10.1002/jrsm.1311

2018-07-30

Research Synthesis Methods

Abstract:Screening references is a time-consuming step necessary for systematic reviews and guideline development. Previous studies have shown that human effort can be reduced by using machine learning software to prioritise large reference collections such that most of the relevant references are identified before screening is completed. We describe and evaluate RobotAnalyst, a Web-based software system that combines text-mining and machine learning algorithms for organising references by their content and actively prioritising them based on a relevancy classification model trained and updated throughout the process. We report an evaluation over 22 reference collections (most are related to public health topics) screened using RobotAnalyst with a total of 43 610 abstract-level decisions. The number of references that needed to be screened to identify 95% of the abstract-level inclusions for the evidence review was reduced on 19 of the 22 collections. Significant gains over random sampling were achieved for all reviews conducted with active prioritisation, as compared with only two of five when prioritisation was not used. RobotAnalyst's descriptive clustering and topic modelling functionalities were also evaluated by public health analysts. Descriptive clustering provided more coherent organisation than topic modelling, and the content of the clusters was apparent to the users across a varying number of clusters. This is the first large-scale study using technology-assisted screening to perform new reviews, and the positive results provide empirical evidence that RobotAnalyst can accelerate the identification of relevant studies. The results also highlight the issue of user complacency and the need for a stopping criterion to realise the work savings.

multidisciplinary sciences,mathematical & computational biology
Outcome-based Evaluation of Systematic Review Automation

Wojciech Kusa,Guido Zuccon,Petr Knoth,Allan Hanbury

DOI: https://doi.org/10.48550/arXiv.2306.17614

2023-06-30

Information Retrieval

Abstract:Current methods of evaluating search strategies and automated citation screening for systematic literature reviews typically rely on counting the number of relevant and not relevant publications. This established practice, however, does not accurately reflect the reality of conducting a systematic review, because not all included publications have the same influence on the final outcome of the systematic review. More specifically, if an important publication gets excluded or included, this might significantly change the overall review outcome, while not including or excluding less influential studies may only have a limited impact. However, in terms of evaluation measures, all inclusion and exclusion decisions are treated equally and, therefore, failing to retrieve publications with little to no impact on the review outcome leads to the same decrease in recall as failing to retrieve crucial publications. We propose a new evaluation framework that takes into account the impact of the reported study on the overall systematic review outcome. We demonstrate the framework by extracting review meta-analysis data and estimating outcome effects using predictions from ranking runs on systematic reviews of interventions from CLEF TAR 2019 shared task. We further measure how closely the obtained outcomes are to the outcomes of the original review if the arbitrary rankings were used. We evaluate 74 runs using the proposed framework and compare the results with those obtained using standard IR measures. We find that accounting for the difference in review outcomes leads to a different assessment of the quality of a system than if traditional evaluation measures were used. Our analysis provides new insights into the evaluation of retrieval results in the context of systematic review automation, emphasising the importance of assessing the usefulness of each document beyond binary relevance.

Endoscopic aspects of the gastric mucosa in relation to its function. I. In the gastritides.

[Number Needed to Be Screened in a Study: a Novel Measure for Disease Screening Effect].

Irreproducibility in Searches of Scientific Literature: A Comparative Analysis.

Increasing the efficiency of study selection for systematic reviews using prioritization tools and a single-screening approach

Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

UN Development Programme and non-communicable diseases

The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching

Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study

The optimal approach for retrieving systematic reviews was achieved when searching MEDLINE and Epistemonikos in addition to reference checking: a methodological validation study

Efficient searching for NICE Public Health Guidelines: would using fewer sources still find the evidence?

How much searching is enough? Comprehensive versus optimal retrieval for technology assessments

Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

An Empirical Assessment of a Systematic Search Process for Systematic Reviews

Abstract 1265: Comparison of mortality- vs. stage-based endpoints in randomized trials of cancer screening: A systematic review with implications for multi-cancer screening trials

158: ASSESSING THE QUALITY OF SEARCH STRATEGIES IN DECISION SUPPORT SYSTEM (DSS) SYSTEMATIC REVIEWS

An evidence-based practice guideline for the peer review of electronic search strategies

Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies

Novel citation-based search method for scientific literature: application to meta-analyses

The contribution of databases to the results of systematic reviews: a cross-sectional study

Prioritising references for systematic reviews with RobotAnalyst: A user study

Outcome-based Evaluation of Systematic Review Automation