Performance of Search Strategies to Retrieve Systematic Reviews of Diagnostic Test Accuracy from the Cochrane Library.

Yuansheng Huang,Zhirong Yang,Jing Wang,Lin Zhuo,Zhixia Li,Siyan Zhan

DOI: https://doi.org/10.1111/jebm.12200

2016-01-01

Journal of Evidence-Based Medicine

Abstract:To compare the performance of search strategies to retrieve systematic reviews of diagnostic test accuracy from The Cochrane Library.

What problem does this paper attempt to address?

Performance Evaluation of Inpatient Service in Beijing: a Horizontal Comparison with Risk Adjustment Based on Diagnosis Related Groups

Weiyan Jian,Yinmin Huang,Mu Hu,Xiumei Zhang

DOI: https://doi.org/10.1186/1472-6963-9-72

2009-01-01

BMC Health Services Research

Abstract:Background: The medical performance evaluation, which provides a basis for rational decision-making, is an important part of medical service research. Current progress with health services reform in China is far from satisfactory, without sufficient regulation. To achieve better progress, an effective tool for evaluating medical performance needs to be established. In view of this, this study attempted to develop such a tool appropriate for the Chinese context.Methods: Data was collected from the front pages of medical records (FPMR) of all large general public hospitals (21 hospitals) in the third and fourth quarter of 2007. Locally developed Diagnosis Related Groups (DRGs) were introduced as a tool for risk adjustment and performance evaluation indicators were established: Charge Efficiency Index (CEI), Time Efficiency Index (TEI) and inpatient mortality of low-risk group cases (IMLRG), to reflect respectively work efficiency and medical service quality. Using these indicators, the inpatient services' performance was horizontally compared among hospitals. Case-mix Index (CMI) was used to adjust efficiency indices and then produce adjusted CEI (aCEI) and adjusted TEI (aTEI). Poisson distribution analysis was used to test the statistical significance of the IMLRG differences between different hospitals.Results: Using the aCEI, aTEI and IMLRG scores for the 21 hospitals, Hospital A and C had relatively good overall performance because their medical charges were lower, LOS shorter and IMLRG smaller. The performance of Hospital P and Q was the worst due to their relatively high charge level, long LOS and high IMLRG. Various performance problems also existed in the other hospitals.Conclusion: It is possible to develop an accurate and easy to run performance evaluation system using Case-Mix as the tool for risk adjustment, choosing indicators close to consumers and managers, and utilizing routine report forms as the basic information source. To keep such a system running effectively, it is necessary to improve the reliability of clinical information and the risk-adjustment ability of Case-Mix.
Evaluating Performance of Local Case-Mix System by International Comparison: a Case Study in Beijing, China.

Wei-Yan Jian,Ming Lu,Tao Cui,Mu Hu

DOI: https://doi.org/10.1002/hpm.1111

2011-01-01

Abstract:Background Case-mix is an important tool for health planning and management in many countries. As a major developing country, China is considering the introduction of the casemix system in the health reform. Beijing, the capital of China, developed a local case-mix version whose performance needs to be evaluated before utilization. Objective The objective of this study was to evaluate the performance of the case-mix system developed in Beijing by comparing it with those used in Australia and the USA. Method A total of 1.3 million inpatient records from 154 hospitals in Beijing in 2008 were grouped respectively using three case-mix systems: ( i) Beijing Diagnosis Related Groups (BJ-DRGs); (ii) US-based All Patient DRGs; and (iii) Australian Refined DRGs. Coefficient of variation (CV) and reduction in variance (RIV) were used to measure the performance of DRGs system. Results The BJ-DRGs produced the best CV and RIV results for expenditure. However, at the level of Major Diagnostic Category (MDC), three MDCs of BJ-DRGs gave the poorest RIVs for both expenditure and length of stay. Conclusions Although the performance of BJ-DRGs was acceptable, further revision and improvement is needed. Comparisons with other mature DRGs versions can assist in identifying the improvement priorities of the local version. Copyright (C) 2011 John Wiley & Sons, Ltd.
Methodological Bias and Variation of Systematic Reviews on Diagnostic Test Accuracy

Z X Li,Z R Yang,X Xiang,P Gao,Z Shu,Y S Huang,Y Cao,F Sun,S Y Zhan

DOI: https://doi.org/10.3760/cma.j.issn.0254-6450.2016.02.027

2016-01-01

Abstract:Objective To analyze methodological bias and variation of systematic reviews on diagnostic test accuracy (DTA).Methods Meta-analyses on DTA were identified through an electronic search through databases as Medline,Embase and Cochrane between 1 January 2008 and 31 December 2012.Results from Meta-analyses on 10 primary studies were included.Pairs of reviewers worked independently to extract the related data of interest,together with those original data of the primary studies.Mixed linear model was used to investigate the direction and strength of the association among the 14 studies,featuring on estimates of the diagnostic accuracy.Results A total of 23 papers on Meta-analyses with 550 primary studies were included.Results from mixed linear model showed that significant low estimates of diagnostic accuracy in studies unsatisfying "the reference standard would likely to correctly classify the target condition" [relative diagnostic odds ration (RDOR) =0.018 6,95％ CI:0.001 0-0.358 5].Studies whose reference standard were not independent of the index test produced significantly higher estimates of diagnostic accuracy (RDOR=2.396 6,95％ CI:1.242 8-4.622 7).Conclusion Messages as "Is the reference standard likely to correctly classify the target condition?" and "Was the reference standard independent of the index test",were the origin of the methodological bias and variation of systematic reviews on diagnostic test accuracy.
158: ASSESSING THE QUALITY OF SEARCH STRATEGIES IN DECISION SUPPORT SYSTEM (DSS) SYSTEMATIC REVIEWS

Zahra Fathifar,Hossein Saeedi Hasankandi,Fatemeh Sadeghi-Ghyassi

DOI: https://doi.org/10.1136/bmjopen-2016-015415.158

IF: 3.006

2017-02-01

BMJ Open

Abstract:Background and aims: Search strategies play a major role in developing a systematic review. A comprehensive literature search dramatically reduces publication bias. The higher sensitive search leads to a more well-prepared systematic review. The aim of this study is evaluating the quality of search strategies in decision support system (DSS) systematic reviews. Methods: A comprehensive literature search using MEDLINE (via Ovid), Embase, Pubmed and Cochrane Library was conducted until September August 2016. The search was limited to systematic reviews and with no language restriction. Meta-reviews were excluded. Only results of searching for “decision support systems” or “decision support system” keywords and limited to title of documents were included in this study. Two independent authors screened the results. Disagreement was resolved by discussion. The search strategies of the included studies were assessed with a nine-question checklist prepared by Faggion et al 1 . Data were extracted into Excel. Results: A total of 47 systematic reviews (SRs) were included. There was a dramatic increase in the number of SRs in recent decade; however no significant improvement was seen in search strategies during the time. The mean score of search strategies among SRs was 4.63 (out of 9). The highest score was 7 (21.2%) and more than 53% of SRs obtained a score between 5 and 7. In 74.4% of SRs, 3–8 databases were searched which 40.4% were considered 4 databases. About 55% of SRs reported search keywords and using Boolean operators that has increased in recent years. Near to 98% of studies were not conducted in duplicate. In near to 75% of studies hand-searching was performed but instead, in 72.3% grey literature was not searched and in 65.9% the authors of studies were not contacted for further relevant information. Fortunately, in about 60% of studies the literature search was performed without language restriction. Conclusions: Lack of duplicate literature searching in systematic reviews was notable. Although most of SRs were searched more than four databases but it seems that in most of them the major databases were not searched and instead, local or specialized databases were considered. Low tendency to grey literature searching was remarkable. It seems that the quality of reporting search strategies in the field of DSS is poor or moderate.

medicine, general & internal
Effects Of Search Strategies On Fault Diagnosis Performance

Q. R. Luo,X. S. Ding,W. W. Qiu,Z. Z. Li

DOI: https://doi.org/10.1109/ieem.2017.8290079

2017-01-01

Abstract:This paper investigates the effects of two diagnostic search strategies (decision table search and hypothesis-and-test search) on fault diagnosis performance, and whether cognitive styles could mediate these effects. A simulated nuclear power plant main control room prototype was employed as a test case for fault diagnosis experiments. Performance was measured by diagnosis accuracy, completion time, frequency of display switching occurrences, mental workload, and self-reported confidence level in the diagnosis task. Results revealed a significant effect of search strategy on the number of display switches, with significantly fewer occurrences for the group using the decision table search strategy as compared to that of the hypothesis-and-test search strategy group. No significant effect of search strategy was observed on the other diagnosis performance measures after controlling for possible effects of cognitive styles. The results suggest a potential advantage of using a decision table search strategy for fault diagnosis given its smaller associated magnitude of display switching.
Protocol for conducting a systematic review on diagnostic accuracy in clinical research

Marco Sguanci,Stefano Mancin,Michela Piredda,Maria Grazia De Marinis

DOI: https://doi.org/10.1016/j.mex.2024.102569

2024-01-20

MethodsX

Abstract:In the landscape of modern medicine, the ability to accurately diagnose various clinical conditions is paramount. As new diagnostic tools continue to emerge, their accuracy must be rigorously assessed before clinical implementation. This paper introduces a systematic review protocol tailored for diagnostic accuracy studies, drawing inspiration from a review on dysphagia screening in post-stroke patients. The protocol, designed with precision and transparency at its core, facilitates a thorough synthesis of evidence, employing tools such as the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) and the Standards for Reporting of Diagnostic Accuracy Studies (STARD) checklist for robust evaluation. The protocol emphasizes registration with the PROSPERO database and adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The systematic search approach encompasses a comprehensive exploration of databases and precise keyword combinations. Distinctive inclusion and exclusion criteria, coupled with a dual-reviewer methodology, ensure the selection of high-quality studies. This framework has the potential to serve as a benchmark for systematic reviews in diagnostic accuracy, highlighting the importance of standardization, transparency, and adaptability in clinical research. This approach paves the way for a research methodology that delves deeper into diagnostic tools across various clinical scenarios, promoting evidence-based advancements in patient care.
A step-by-step guide to the systematic review and meta-analysis of diagnostic and prognostic test accuracy evaluations

Z Liu,Z Yao,C Li,X Liu,H Chen,C Gao

DOI: https://doi.org/10.1038/bjc.2013.185

IF: 9.075

2013-05-21

British Journal of Cancer

Abstract:In evidence-based medicine (EBM), systematic reviews and meta-analyses have been widely applied in biological and medical research. Moreover, the most popular application of meta-analyses in this field may be to examine diagnostic (sensitivity and specificity) and prognostic (hazard ratio (HR) and its variance, standard error (SE) or confidence interval (CI)) test accuracy. However, conducting such analyses requires not only a great deal of time but also an advanced professional knowledge of mathematics, statistics and computer science. Regarding the practical application of meta-analyses for diagnostic and prognostic markers, the majority of users are clinicians and biologists, most of whom are not skilled at mathematics and computer science in particular. Hence, it is necessary for these users to have a simplified version of a protocol to help them to quickly conduct meta-analyses of the accuracy of diagnostic and prognostic tests. The aim of this paper is to enable individuals who have never performed a meta-analysis to do so from scratch. The paper does not attempt to serve as a comprehensive theoretical guide but instead describes one rigorous way of conducting a meta-analysis for diagnostic and prognostic markers. Investigators who follow the outlined methods should be able to understand the basic ideas behind the steps taken, the meaning of the meta-analysis results obtained for diagnostic and prognostic markers and the scope of questions that can be answered with Systematic Reviews and Meta-Analyses (SRMA). The presented protocols have been successfully tested by clinicians without meta-analysis experience.

oncology
Copas-like selection model to correct publication bias in systematic review of diagnostic test studies

Jin Piao,Yulun Liu,Yong Chen,Jing Ning

DOI: https://doi.org/10.1177/0962280218791602

IF: 2.494

2018-07-31

Statistical Methods in Medical Research

Abstract:The accuracy of a diagnostic test, which is often quantified by a pair of measures such as sensitivity and specificity, is critical for medical decision making. Separate studies of an investigational diagnostic test can be combined through meta-analysis; however, such an analysis can be threatened by publication bias. To the best of our knowledge, there is no existing method that accounts for publication bias in the meta-analysis of diagnostic tests involving bivariate outcomes. In this paper, we extend the Copas selection model from univariate outcomes to bivariate outcomes for the correction of publication bias when the probability of a study being published can depend on its sensitivity, specificity, and the associated standard errors. We develop an expectation-maximization algorithm for the maximum likelihood estimation under the proposed selection model. We investigate the finite sample performance of the proposed method through simulation studies and illustrate the method by assessing a meta-analysis of 17 published studies of a rapid diagnostic test for influenza.

health care sciences & services,medical informatics,mathematical & computational biology,statistics & probability
Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol

Yuelun Zhang,Siyu Liang,Yunying Feng,Qing Wang,Feng Sun,Shi Chen,Yiying Yang,Xin He,Huijuan Zhu,Hui Pan

DOI: https://doi.org/10.1186/s13643-021-01881-5

2022-01-15

Systematic Reviews

Abstract:Abstract Background Systematic review is an indispensable tool for optimal evidence collection and evaluation in evidence-based medicine. However, the explosive increase of the original literatures makes it difficult to accomplish critical appraisal and regular update. Artificial intelligence (AI) algorithms have been applied to automate the literature screening procedure in medical systematic reviews. In these studies, different algorithms were used and results with great variance were reported. It is therefore imperative to systematically review and analyse the developed automatic methods for literature screening and their effectiveness reported in current studies. Methods An electronic search will be conducted using PubMed, Embase, ACM Digital Library, and IEEE Xplore Digital Library databases, as well as literatures found through supplementary search in Google scholar, on automatic methods for literature screening in systematic reviews. Two reviewers will independently conduct the primary screening of the articles and data extraction, in which nonconformities will be solved by discussion with a methodologist. Data will be extracted from eligible studies, including the basic characteristics of study, the information of training set and validation set, and the function and performance of AI algorithms, and summarised in a table. The risk of bias and applicability of the eligible studies will be assessed by the two reviewers independently based on Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Quantitative analyses, if appropriate, will also be performed. Discussion Automating systematic review process is of great help in reducing workload in evidence-based practice. Results from this systematic review will provide essential summary of the current development of AI algorithms for automatic literature screening in medical evidence synthesis and help to inspire further studies in this field. Systematic review registration PROSPERO CRD42020170815 (28 April 2020).

medicine, general & internal
Reporting of Search Strategy Among Systematic Reviews in Leading Dental Specialty Journals: A Research‐On‐Research Study

Jiayi Li,Xueqian Yu,Lingyue Gao,Lijing Xin,Yan Wang,Feiyang Guo,Fang Hua

DOI: https://doi.org/10.1111/joor.13904

2024-11-28

Journal of Oral Rehabilitation

Abstract:Background Literature searching is one of the main determinants of a systematic review (SR)'s reliability. Thus, adequate reporting of search strategy is essential for the critical appraisal of SRs and evidence‐based practice. Objectives To assess the reporting quality of search strategy among SRs in leading dental specialty journals, and to identify factors associated with quality of reporting. Materials and Methods Six leading dental journals with the highest 5‐year impact factors in their respective specialty were included. A hand search was undertaken to identify SRs published between 2017 and 2022. Full texts were reviewed by two authors to identify eligible SRs. Reporting quality was assessed and scored using a modified 15‐item checklist based on Preferred Reporting Items for Systematic reviews and Meta‐Analyses literature search extension (PRISMA‐S). Univariable and multivariable linear regression analyses were then performed. Settings and Sample population A total of 152 reviews were included and assessed. Results As for information sources, only 'citation searching' was adequately reported in most reviews 110 (72.4%). Only 23 (15.1%) of the included reviews clearly reported search strategies. Information about peer review was provided in only 10 reviews (6.6%). Only 91 (59.9%) of the included reviews documented the total records clearly. According to multivariable regression analysis, industrial funding (p = 0.012), registration (p = 0.013) and librarian involvement (p = 0.004) were significantly associated with higher reporting quality. Conclusions The reporting quality of search strategy among SRs in leading dental specialty journals is suboptimal. Researchers, librarians, reviewers and journal editors in dentistry need to be familiar with the PRISMA‐S checklist, and make concerted efforts to improve the reporting of search strategy in SRs.

dentistry, oral surgery & medicine
Performance Evaluation of a Hospital based on Diagnosis Related Groups

Juan Zhang,Feng Yu,Li Li

DOI: https://doi.org/10.3969/j.issn.1672-2566.2017.03.005

2017-01-01

Abstract:Objectives To conduct a comprehensive evaluation on the hospital service performance of the major diagnostic category in the study of the whole hospital by using the diagnosis related groups, and then put forward advices on how to strengthen the hospital diagnosis and treatment ability of main diseases as well as improve inpatient service performance.Methods To conduct analysis on the discharged patients of a hospital from 2013 to 2015 with the application of indexes included case-mixindex(CMI), DRGs number, time-consuming index, cost-consuming index, mortality rate of low-risk group according to BJ-DRGs packet mode. Results From 2013 to 2015, the main MDC changes were not large, the overall DRGs number gradually was increased, the cost-consuming index was gradually decreased, the time-consuming index was increased, CMI change was not obvious, the low risk of mortality was increased.Conclusions Through the research of the hospital in recent three years, we could treat most of the musculoskeletal system diseases and had wide range of medical services. We should maintain the advantage, and strengthen the development of other departments at the same time so as to reflect the positioning of the general hospital. And also, we should improve the efficiency of medical services and pay attention to the security of medical services.
On the performance of hybrid search strategies for systematic literature reviews in software engineering

Erica Mourão,João Felipe Pimentel,Leonardo Murta,Marcos Kalinowski,Emilia Mendes,Claes Wohlin

DOI: https://doi.org/10.1016/j.infsof.2020.106294

IF: 3.9

2020-07-01

Information and Software Technology

Abstract:<h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Context</h3><p>When conducting a Systematic Literature Review (SLR), researchers usually face the challenge of designing a search strategy that appropriately balances result quality and review effort. Using digital library (or database) searches or snowballing alone may not be enough to achieve high-quality results. On the other hand, using both digital library searches and snowballing together may increase the overall review effort.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Objective</h3><p>The goal of this research is to propose and evaluate hybrid search strategies that selectively combine database searches with snowballing.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Method</h3><p>We propose four hybrid search strategies combining database searches in digital libraries with iterative, parallel, or sequential backward and forward snowballing. We simulated the strategies over three existing SLRs in SE that adopted both database searches and snowballing. We compared the outcome of digital library searches, snowballing, and hybrid strategies using precision, recall, and F-measure to investigate the performance of each strategy.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Results</h3><p>Our results show that, for the analyzed SLRs, combining database searches from the Scopus digital library with parallel or sequential snowballing achieved the most appropriate balance of precision and recall.</p><h3 class="u-h4 u-margin-m-top u-margin-xs-bottom">Conclusion</h3><p>We put forward that, depending on the goals of the SLR and the available resources, using a hybrid search strategy involving a representative digital library and parallel or sequential snowballing tends to represent an appropriate alternative to be used when searching for evidence in SLRs.</p>

computer science, information systems, software engineering
A meta-analysis for the diagnostic accuracy of SelectMDx in prostate cancer

Hanting Wu,Yanling Wu,Peijie He,Juan Liang,Xiujuan Xu,Conghua Ji

DOI: https://doi.org/10.1371/journal.pone.0285745

IF: 3.7

2024-02-08

PLoS ONE

Abstract:To overview the diagnostic accuracy of SelectMDx for the detection of clinically significant prostate cancer and to review sources of methodologic variability. Four electronic databases, including PubMed, Embase, Web of Science, and Cochrane Library were searched for eligible studies investigating the diagnostic value of SelectMDx compared with the gold standard. The pooled sensitivity, specificity, and positive and negative predictive values were calculated. Included studies were assessed according to the Standards for Quality Assessment of Diagnostic Accuracy Studies 2 tool. The review identified 14 relevant publications with 2579 patients. All reports constituted phase 1 biomarker studies. Pooled analysis of findings found an area under the receiver operating characteristic analysis curve of 70% [95% CI, 66%-74%], a sensitivity of 81% [95% CI, 69%-89%], and a specificity of 52% [95% CI, 41%-63%]. The positive likelihood ratio was 1.68, and the negative predictive value is 0.37. Factors that may influence variability in test results included the breath collection method, the patient’s physiologic condition, the test environment, and the method of analysis. Considerable heterogeneity was observed among the studies owing to the difference in the sample size. SelectMDx appears to have moderate to good diagnostic accuracy in differentiating patients with clinically significant prostate cancer from people at high risk of developing prostate cancer. Higher-quality clinical studies assessing the diagnostic accuracy of SelectMDx for clinically significant cancer are still needed.

multidisciplinary sciences
A Study on the Index System of the Performance Assessment for the Public Secondary-Level General Hospitals in Shanghai

金其林,钱丽君,陈永聪,薛迪,付晨,陈雯,韩菁,王颖丽

DOI: https://doi.org/10.3969/j.issn.1671-0592.2010.03.012

2010-01-01

Abstract:An index system of the performance assessment for the public secondary-level general hospitals in Shanghai was designed with the methods of literature review,data collection,expert consultation and performance assessment simulation. According to the results of the simulated performance assessment in the public secondary-level general hospitals in Shanghai,the different criteria for performance classification were also made for the different types of the public secondary-level general hospitals in Shanghai.
A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization

Sihan Li,Naiwen Bian,Zhenyu Chen,Dongjiang You,Yuchen He

DOI: https://doi.org/10.1109/QSIC.2010.15

2010-01-01

Abstract:Test case prioritization is an approach aiming at increasing the rate of faults detection during the testing phase, by reordering test case execution. Many techniques for regression test case prioritization have been proposed. In this paper, we perform a simulation experiment to study five search algorithms for test case prioritization and compare the performance of these algorithms. The target of the study is to have an in-depth investigation and improve the generality of the comparison results. The simulation study provides two useful guidelines: (1) Two search algorithms, Additional Greedy Algorithm and 2-Optimal Greedy Algorithm, outperform the other three search algorithms in most cases. (2) The performance of the five search algorithms will be affected by the overlap of test cases with regard to test requirements.
Correlations Between Human Performance in Information Seeking, Information Integration, and Overall Process in Diagnostic Tasks

Xi Lyu,Zhizhong Li

DOI: https://doi.org/10.1080/10447318.2019.1631545

IF: 4.92

2019-01-01

International Journal of Human-Computer Interaction

Abstract:Insufficient effort has been made to investigate details of the diagnostic process, specifically information seeking and information integration. In this study, participants were asked to accomplish eight diagnostic tasks of locating the leaking position of some working material in a simulated nuclear power plant. The participants' information seeking actions and possibility judgments about all potential leaking positions were recorded and then described using eight dependent variables concerning their gains and expenses. Generalized linear mixed-models were used to evaluate the effects of time pressure and the correlations. The results showed that (1) information seeking and information integration were closely correlated in multifaceted ways; (2) higher diagnostic accuracy was associated with fewer information acquiring actions but with longer average intervals between these actions; and (3) high time-pressure induced speed-and-accuracy trade-offs in the sub-processes, and the overall diagnostic speed and accuracy were negatively related between individuals, but positively related between trials
Statistical Methods for Comparing Predictive Values in Medical Diagnosis

Chanrim Park,Seo Young Park,Hwa Jung Kim,Hee Jung Shin

DOI: https://doi.org/10.3348/kjr.2024.0049

Abstract:Evaluating the performance of a binary diagnostic test, including artificial intelligence classification algorithms, involves measuring sensitivity, specificity, positive predictive value, and negative predictive value. Particularly when comparing the performance of two diagnostic tests applied on the same set of patients, these metrics are crucial for identifying the more accurate test. However, comparing predictive values presents statistical challenges because their denominators depend on the test outcomes, unlike the comparison of sensitivities and specificities. This paper reviews existing methods for comparing predictive values and proposes using the permutation test. The permutation test is an intuitive, non-parametric method suitable for datasets with small sample sizes. We demonstrate each method using a dataset from MRI and combined modality of mammography and ultrasound in diagnosing breast cancer.
Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis

Yunying Feng,Siyu Liang,Yuelun Zhang,Shi Chen,Qing Wang,Tianze Huang,Feng Sun,Xiaoqing Liu,Huijuan Zhu,Hui Pan

DOI: https://doi.org/10.1093/jamia/ocac066

2022-05-31

Journal of the American Medical Informatics Association

Abstract:Abstract Objective We aim to investigate the application and accuracy of artificial intelligence (AI) methods for automated medical literature screening for systematic reviews. Materials and Methods We systematically searched PubMed, Embase, and IEEE Xplore Digital Library to identify potentially relevant studies. We included studies in automated literature screening that reported study question, source of dataset, and developed algorithm models for literature screening. The literature screening results by human investigators were considered to be the reference standard. Quantitative synthesis of the accuracy was conducted using a bivariate model. Results Eighty-six studies were included in our systematic review and 17 studies were further included for meta-analysis. The combined recall, specificity, and precision were 0.928 [95% confidence interval (CI), 0.878–0.958], 0.647 (95% CI, 0.442–0.809), and 0.200 (95% CI, 0.135–0.287) when achieving maximized recall, but were 0.708 (95% CI, 0.570–0.816), 0.921 (95% CI, 0.824–0.967), and 0.461 (95% CI, 0.375–0.549) when achieving maximized precision in the AI models. No significant difference was found in recall among subgroup analyses including the algorithms, the number of screened literatures, and the fraction of included literatures. Discussion and Conclusion This systematic review and meta-analysis study showed that the recall is more important than the specificity or precision in literature screening, and a recall over 0.95 should be prioritized. We recommend to report the effectiveness indices of automatic algorithms separately. At the current stage manual literature screening is still indispensable for medical systematic reviews.

information science & library science,computer science, information systems, interdisciplinary applications,health care sciences & services,medical informatics
Performance of Self-Performed SARS-CoV-2 Rapid Antigen Test: a Systematic Review and Meta-Analysis

Peiling Cai,Junren Wang,Peng Ye,Yarong Zhang,Mengping Wang,Ronglian Guo,Hongying Zhao

DOI: https://doi.org/10.3389/fpubh.2024.1402949

IF: 5.2

2024-01-01

Frontiers in Public Health

Abstract:BackgroundThe aim of this study was to investigate the accuracy of self-tested SARS-CoV-2 rapid antigen tests.MethodsDatabases of Pubmed, Embase, and Cochrane Library were searched for original studies investigating accuracy of self-tested SARS-CoV-2 rapid antigen tests, with RT-PCR as “gold standard.”ResultsForty-five eligible studies were found after database searching and screening using pre-defined criteria. The accuracy results from 50,897 suspected COVID-19 patients were pooled, and the overall sensitivity, specificity and diagnostic odds ratio were 0.77, 1.00, and 625.95, respectively. Subgroup analysis showed higher sensitivity of rapid antigen tests in subgroups of Abbott Panbio, self-collected nasal swab samples, and use of nasopharyngeal or oropharyngeal swab and lower Ct cutoff value in RT-PCR.ConclusionFully self-performed SARS-CoV-2 rapid antigen tests showed overall high accuracy compared to “gold standard,” and are reliable surrogates for the standard test of COVID-19 using nasopharyngeal or oropharyngeal samples and RT-PCR.
Evaluating the performance of three different cervical cancer screening modalities in a large prospective population-based cohort.

Fangbin Song,Hui Du,Aimin Xiao,Chun Wang,Xia Huang,Zhihong Liu,Meifang Zhao,Hongjian Men,Ruifang Wu

DOI: https://doi.org/10.1016/j.jiph.2020.08.008

IF: 7.537

2020-01-01

Journal of Infection and Public Health

Abstract:BACKGROUND:This study aimed to evaluate three different patterns of cervical cancer screening strategies for detection of cervical diseases. METHODS:In total, 10,186 women aged 21-70 years attending cervical screening program were recruited and cotested by human papillomavirus (HPV) assays and cytology. Three-year histological follow-up data was recorded on women with abnormal screening results, and six clinically common screening algorithms were evaluated. RESULTS:Significantly better protection against cervical intraepithelial neoplasia 2 or worse (CIN2+) at three-year follow-up was associated with a negative HPV result than by normal cytology at baseline. HPV screening was more sensitive and less specific than cytology screening. Moreover, HPV screening with HPV16/18 and reflex cytology (atypical squamous cells of undetermined significance [ASCUS] threshold) showed a similar sensitivity (94.6% vs. 98.2%, p = 0.125) and a superior specificity as compared to cotesting reflex HPV16/18 and cytology (ASCUS threshold) for CIN2+ (95.8% vs. 95.1%, p < 0.0001), achieving a colposcopy referral rate of 5.4%, and consuming 4.8 colposcopies and 4.4 cytology tests to find one CIN2+. CONCLUSIONS:HPV screening with triage of HPV-positive women by HPV16/18 genotyping and cytology provided a good equilibrium between screening effectiveness, the number of cytology tests required, and referral rates; HPV testing was similar in sensitivity to cotesting and safer than cytology, thus especially suitable for large population-based screening programs.

Performance of Search Strategies to Retrieve Systematic Reviews of Diagnostic Test Accuracy from the Cochrane Library.

Performance Evaluation of Inpatient Service in Beijing: a Horizontal Comparison with Risk Adjustment Based on Diagnosis Related Groups

Evaluating Performance of Local Case-Mix System by International Comparison: a Case Study in Beijing, China.

Methodological Bias and Variation of Systematic Reviews on Diagnostic Test Accuracy

158: ASSESSING THE QUALITY OF SEARCH STRATEGIES IN DECISION SUPPORT SYSTEM (DSS) SYSTEMATIC REVIEWS

Effects Of Search Strategies On Fault Diagnosis Performance

Protocol for conducting a systematic review on diagnostic accuracy in clinical research

A step-by-step guide to the systematic review and meta-analysis of diagnostic and prognostic test accuracy evaluations

Copas-like selection model to correct publication bias in systematic review of diagnostic test studies

Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol

Reporting of Search Strategy Among Systematic Reviews in Leading Dental Specialty Journals: A Research‐On‐Research Study

Performance Evaluation of a Hospital based on Diagnosis Related Groups

On the performance of hybrid search strategies for systematic literature reviews in software engineering

A meta-analysis for the diagnostic accuracy of SelectMDx in prostate cancer

A Study on the Index System of the Performance Assessment for the Public Secondary-Level General Hospitals in Shanghai

A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization

Correlations Between Human Performance in Information Seeking, Information Integration, and Overall Process in Diagnostic Tasks

Statistical Methods for Comparing Predictive Values in Medical Diagnosis

Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis

Performance of Self-Performed SARS-CoV-2 Rapid Antigen Test: a Systematic Review and Meta-Analysis

Evaluating the performance of three different cervical cancer screening modalities in a large prospective population-based cohort.