ModelEd, TestEd, TrustEd

J. Katoen,R. Langerak,A. Rensink

DOI: https://doi.org/10.1007/978-3-319-68270-9

Abstract:

What problem does this paper attempt to address?

APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support

Jethro C C Kwong,Adree Khondker,Katherine Lajkosz,Matthew B A McDermott,Xavier Borrat Frigola,Melissa D McCradden,Muhammad Mamdani,Girish S Kulkarni,Alistair E W Johnson

DOI: https://doi.org/10.1001/jamanetworkopen.2023.35377

2023-09-05

Abstract:Importance: Artificial intelligence (AI) has gained considerable attention in health care, yet concerns have been raised around appropriate methods and fairness. Current AI reporting guidelines do not provide a means of quantifying overall quality of AI research, limiting their ability to compare models addressing the same clinical question. Objective: To develop a tool (APPRAISE-AI) to evaluate the methodological and reporting quality of AI prediction models for clinical decision support. Design, setting, and participants: This quality improvement study evaluated AI studies in the model development, silent, and clinical trial phases using the APPRAISE-AI tool, a quantitative method for evaluating quality of AI studies across 6 domains: clinical relevance, data quality, methodological conduct, robustness of results, reporting quality, and reproducibility. These domains included 24 items with a maximum overall score of 100 points. Points were assigned to each item, with higher points indicating stronger methodological or reporting quality. The tool was applied to a systematic review on machine learning to estimate sepsis that included articles published until September 13, 2019. Data analysis was performed from September to December 2022. Main outcomes and measures: The primary outcomes were interrater and intrarater reliability and the correlation between APPRAISE-AI scores and expert scores, 3-year citation rate, number of Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) low risk-of-bias domains, and overall adherence to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement. Results: A total of 28 studies were included. Overall APPRAISE-AI scores ranged from 33 (low quality) to 67 (high quality). Most studies were moderate quality. The 5 lowest scoring items included source of data, sample size calculation, bias assessment, error analysis, and transparency. Overall APPRAISE-AI scores were associated with expert scores (Spearman ρ, 0.82; 95% CI, 0.64-0.91; P < .001), 3-year citation rate (Spearman ρ, 0.69; 95% CI, 0.43-0.85; P < .001), number of QUADAS-2 low risk-of-bias domains (Spearman ρ, 0.56; 95% CI, 0.24-0.77; P = .002), and adherence to the TRIPOD statement (Spearman ρ, 0.87; 95% CI, 0.73-0.94; P < .001). Intraclass correlation coefficient ranges for interrater and intrarater reliability were 0.74 to 1.00 for individual items, 0.81 to 0.99 for individual domains, and 0.91 to 0.98 for overall scores. Conclusions and relevance: In this quality improvement study, APPRAISE-AI demonstrated strong interrater and intrarater reliability and correlated well with several study quality measures. This tool may provide a quantitative approach for investigators, reviewers, editors, and funding organizations to compare the research quality across AI studies for clinical decision support.
[Eosinophilic globules in the Spitz nevus].

A. Moreno,N. Salvatella,J. de Moragas

Abstract:We compared the incidence and morphologic appearance of eosinophilic globules in Spitz's nevi, malignant melanoma and compound nevi. Eosinophilic globules were frequently found in Spitz's nevi, specially in those with junctional or compound features. An statistical significant increase of eosinophilic globules was found in Spitz's nevi when compared with malignant melanoma. Eosinophilic globules are not an reliable criteria for a diagnosis of a bening lesion. The use of multiple histologic parameters for differential diagnosis is necessary.
[Risk on Bias Assessment: (6) A Revised Tool for the Quality Assessment on Diagnostic Accuracy Studies (QUADAS-2)].

Y J Qu,Z R Yang,F Sun,S Y Zhan

DOI: https://doi.org/10.3760/cma.j.issn.0254-6450.2018.04.028

2018-01-01

Abstract:This paper introduced the Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2), including the development and comparison with the original QUADAS, and illustrated the application of QUADAS-2 in a published paper related to the study on diagnostic accuracy which was included in systematic review and Meta-analysis. QUADAS-2 presented considerable improvement over the original tool. Confused items that included in QUADAS had disappeared and the quality assessment of the original study replaced by the rating of risk on bias and applicability. This was implemented through the description on the four main domains with minimal overlapping and answering the signal questions in each domain. The risk of bias and applicability with 'high','low' or 'unclear' was in line with the risk of bias assessment of intervention studies in Cochrane, so to replace the total score of quality assessment in QUADAS. Meanwhile, QUADAS-2 was also applicable to assess the diagnostic accuracy studies in which follow-up without prognosis was involved in golden standard. It was useful to assess the overall methodological quality of the study despite more time consuming than the original QUADAS. However, QUADAS-2 needs to be modified to apply in comparative studies on diagnostic accuracy and we hope the users would follow the updates and give their feedbacks on line.
Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

Viknesh Sounderajah,Hutan Ashrafian,Robert M Golub,Shravya Shetty,Jeffrey De Fauw,Lotty Hooft,Karel Moons,Gary Collins,David Moher,Patrick M Bossuyt,Ara Darzi,Alan Karthikesalingam,Alastair K Denniston,Bilal Akhter Mateen,Daniel Ting,Darren Treanor,Dominic King,Felix Greaves,Jonathan Godwin,Jonathan Pearson-Stuttard,Leanne Harling,Matthew McInnes,Nader Rifai,Nenad Tomasev,Pasha Normahani,Penny Whiting,Ravi Aggarwal,Sebastian Vollmer,Sheraz R Markar,Trishan Panch,Xiaoxuan Liu

DOI: https://doi.org/10.1136/bmjopen-2020-047709

IF: 3.006

2021-06-01

BMJ Open

Abstract:Introduction Standards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI. Methods and analysis The development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group’s efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption. Ethics and dissemination Ethical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

medicine, general & internal
Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms

Luigi Angelo Vaira,Jerome R. Lechien,Vincenzo Abbate,Fabiana Allevi,Giovanni Audino,Giada Anna Beltramini,Michela Bergonzani,Paolo Boscolo-Rizzo,Gianluigi Califano,Giovanni Cammaroto,Carlos M. Chiesa-Estomba,Umberto Committeri,Salvatore Crimi,Nicholas R. Curran,Francesco di Bello,Arianna di Stadio,Andrea Frosolini,Guido Gabriele,Isabelle M. Gengler,Fabio Lonardi,Fabio Maglitto,Miguel Mayo-Yáñez,Marzia Petrocelli,Resi Pucci,Alberto Maria Saibene,Gianmarco Saponaro,Alessandro Tel,Franco Trabalzini,Eleonora M. C. Trecca,Valentino Vellone,Giovanni Salzano,Giacomo De Riu

DOI: https://doi.org/10.1007/s00405-024-08710-0

2024-05-04

European Archives of Oto-Rhino-Laryngology

Abstract:Abstract Background The widespread diffusion of Artificial Intelligence (AI) platforms is revolutionizing how health-related information is disseminated, thereby highlighting the need for tools to evaluate the quality of such information. This study aimed to propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms. Methods The QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed by 27 reviewers from 25 academic centers worldwide. Construct validity, internal consistency, inter-rater and test–retest reliability were assessed to validate the tool. Results The validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach's alpha = 0.837). The Interclass Correlation Coefficient was 0.983 (95% CI 0.973–0.991; F (29,542) = 68.3; p < 0.001), indicating excellent reliability. Test–retest reliability analysis revealed a moderate-to-strong correlation with a Pearson’s coefficient of 0.876 (95% CI 0.859–0.891; p < 0.001). Conclusions The QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.

otorhinolaryngology
Stakeholder perspectives towards diagnostic artificial intelligence: a co-produced qualitative evidence synthesis

Rachel Yi Ling Kuo,Alexander Freethy,Judi Smith,Rosie Hill,Joanna C,Derek Jerome,Eli Harriss,Gary S Collins,Elizabeth Tutton,Dominic Furniss

DOI: https://doi.org/10.1016/j.eclinm.2024.102555

IF: 15.1

2024-03-22

EClinicalMedicine

Abstract:Background: Diagnosis is a cornerstone of medical practice. Worldwide, there is increased demand for diagnostic services, exacerbating workforce shortages. Artificial intelligence (AI) technologies may improve diagnostic efficiency, accuracy, and access. Understanding stakeholder perspectives is key to informing implementation of complex interventions. We systematically reviewed the literature on stakeholder perspectives on diagnostic AI, including all English-language peer-reviewed primary qualitative or mixed-methods research. Methods: We searched PubMed, Ovid MEDLINE/Embase, Scopus, CINAHL and Web of Science (22/2/2023 and updated 8/2/2024). The Critical Appraisal Skills Programme Checklist informed critical appraisal. We used a 'best-fit' framework approach for analysis, using the Non-adoption, Abandonment, Scale-up, Spread, Sustainability (NASSS) framework. This study was pre-registered (PROSPERO CRD42022313782). Findings: We screened 16,577 articles and included 44. 689 participants were interviewed, and 402 participated in focus groups. Four stakeholder groups were described: patients, clinicians, researchers and healthcare leaders. We found an under-representation of patients, researchers and leaders across articles. We summarise the differences and relationships between each group in a conceptual model, hinging on the establishment of trust, engagement and collaboration. We present a modification of the NASSS framework, tailored to diagnostic AI. Interpretation: We provide guidance for future research and implementation of diagnostic AI, highlighting the importance of representing all stakeholder groups. We suggest that implementation strategies consider how any proposed software fits within the extended NASSS-AI framework, and how stakeholder priorities and concerns have been addressed. Funding: RK is supported by an NIHR Doctoral Research Fellowship grant (NIHR302562), which funded patient and public involvement activities, and access to Covidence.
Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review

Ying Wang,Nian Li,Lingmin Chen,Miaomiao Wu,Sha Meng,Zelei Dai,Yonggang Zhang,Mike Clarke

DOI: https://doi.org/10.2196/46089

IF: 7.076

2023-11-22

Journal of Medical Internet Research

Abstract:Background The application of artificial intelligence (AI) in the delivery of health care is a promising area, and guidelines, consensus statements, and standards on AI regarding various topics have been developed. Objective We performed this study to assess the quality of guidelines, consensus statements, and standards in the field of AI for medicine and to provide a foundation for recommendations about the future development of AI guidelines. Methods We searched 7 electronic databases from database establishment to April 6, 2022, and screened articles involving AI guidelines, consensus statements, and standards for eligibility. The AGREE II (Appraisal of Guidelines for Research & Evaluation II) and RIGHT (Reporting Items for Practice Guidelines in Healthcare) tools were used to assess the methodological and reporting quality of the included articles. Results This systematic review included 19 guideline articles, 14 consensus statement articles, and 3 standard articles published between 2019 and 2022. Their content involved disease screening, diagnosis, and treatment; AI intervention trial reporting; AI imaging development and collaboration; AI data application; and AI ethics governance and applications. Our quality assessment revealed that the average overall AGREE II score was 4.0 (range 2.2-5.5; 7-point Likert scale) and the mean overall reporting rate of the RIGHT tool was 49.4% (range 25.7%-77.1%). Conclusions The results indicated important differences in the quality of different AI guidelines, consensus statements, and standards. We made recommendations for improving their methodological and reporting quality. Trial Registration PROSPERO International Prospective Register of Systematic Reviews (CRD42022321360); https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=321360

health care sciences & services,medical informatics
A Comprehensive, Valid, and Reliable Tool to Assess the Degree of Responsibility of Digital Health Solutions That Operate With or Without Artificial Intelligence: 3-Phase Mixed Methods Study

Pascale Lehoux,Robson Rocha de Oliveira,Lysanne Rivard,Hudson Pacifico Silva,Hassane Alami,Carl Maria Mörch,Kathy Malas

DOI: https://doi.org/10.2196/48496

IF: 7.076

2023-08-29

Journal of Medical Internet Research

Abstract:Background: Clinicians' scope of responsibilities is being steadily transformed by digital health solutions that operate with or without artificial intelligence (DAI solutions). Most tools developed to foster ethical practices lack rigor and do not concurrently capture the health, social, economic, and environmental issues that such solutions raise. Objective: To support clinical leadership in this field, we aimed to develop a comprehensive, valid, and reliable tool that measures the responsibility of DAI solutions by adapting the multidimensional and already validated Responsible Innovation in Health Tool. Methods: We conducted a 3-phase mixed methods study. Relying on a scoping review of available tools, phase 1 (concept mapping) led to a preliminary version of the Responsible DAI solutions Assessment Tool. In phase 2, an international 2-round e-Delphi expert panel rated on a 5-level scale the importance, clarity, and appropriateness of the tool's components. In phase 3, a total of 2 raters independently applied the revised tool to a sample of DAI solutions (n=25), interrater reliability was measured, and final minor changes were made to the tool. Results: The mapping process identified a comprehensive set of responsibility premises, screening criteria, and assessment attributes specific to DAI solutions. e-Delphi experts critically assessed these new components and provided comments to increase content validity (n=293), and after round 2, consensus was reached on 85% (22/26) of the items surveyed. Interrater agreement was substantial for a subcriterion and almost perfect for all other criteria and assessment attributes. Conclusions: The Responsible DAI solutions Assessment Tool offers a comprehensive, valid, and reliable means of assessing the degree of responsibility of DAI solutions in health. As regulation remains limited, this forward-looking tool has the potential to change practice toward more equitable as well as economically and environmentally sustainable digital health care.

health care sciences & services,medical informatics
Detecting Algorithmic Errors and Patient Harms for AI-Enabled Medical Devices in Randomized Controlled Trials: Protocol for a Systematic Review

Aditya U Kale,Henry David Jeffry Hogg,Russell Pearson,Ben Glocker,Su Golder,April Coombe,Justin Waring,Xiaoxuan Liu,David J Moore,Alastair K Denniston

DOI: https://doi.org/10.2196/51614

2024-06-28

Abstract:Background: Artificial intelligence (AI) medical devices have the potential to transform existing clinical workflows and ultimately improve patient outcomes. AI medical devices have shown potential for a range of clinical tasks such as diagnostics, prognostics, and therapeutic decision-making such as drug dosing. There is, however, an urgent need to ensure that these technologies remain safe for all populations. Recent literature demonstrates the need for rigorous performance error analysis to identify issues such as algorithmic encoding of spurious correlations (eg, protected characteristics) or specific failure modes that may lead to patient harm. Guidelines for reporting on studies that evaluate AI medical devices require the mention of performance error analysis; however, there is still a lack of understanding around how performance errors should be analyzed in clinical studies, and what harms authors should aim to detect and report. Objective: This systematic review will assess the frequency and severity of AI errors and adverse events (AEs) in randomized controlled trials (RCTs) investigating AI medical devices as interventions in clinical settings. The review will also explore how performance errors are analyzed including whether the analysis includes the investigation of subgroup-level outcomes. Methods: This systematic review will identify and select RCTs assessing AI medical devices. Search strategies will be deployed in MEDLINE (Ovid), Embase (Ovid), Cochrane CENTRAL, and clinical trial registries to identify relevant papers. RCTs identified in bibliographic databases will be cross-referenced with clinical trial registries. The primary outcomes of interest are the frequency and severity of AI errors, patient harms, and reported AEs. Quality assessment of RCTs will be based on version 2 of the Cochrane risk-of-bias tool (RoB2). Data analysis will include a comparison of error rates and patient harms between study arms, and a meta-analysis of the rates of patient harm in control versus intervention arms will be conducted if appropriate. Results: The project was registered on PROSPERO in February 2023. Preliminary searches have been completed and the search strategy has been designed in consultation with an information specialist and methodologist. Title and abstract screening started in September 2023. Full-text screening is ongoing and data collection and analysis began in April 2024. Conclusions: Evaluations of AI medical devices have shown promising results; however, reporting of studies has been variable. Detection, analysis, and reporting of performance errors and patient harms is vital to robustly assess the safety of AI medical devices in RCTs. Scoping searches have illustrated that the reporting of harms is variable, often with no mention of AEs. The findings of this systematic review will identify the frequency and severity of AI performance errors and patient harms and generate insights into how errors should be analyzed to account for both overall and subgroup performance. Trial registration: PROSPERO CRD42023387747; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=387747. International registered report identifier (irrid): PRR1-10.2196/51614.
Use of artificial intelligence in obstetric and gynaecological diagnostics: a protocol for a systematic review and meta-analysis

Anjalee Chaurasia,Georgia Curry,Yi Zhao,Fatema Dawoodbhoy,Jennifer Green,Matilde Vaninetti,Nishel Shah,Orene Greer

DOI: https://doi.org/10.1136/bmjopen-2023-082287

IF: 3.006

2024-05-09

BMJ Open

Abstract:Introduction Emerging developments in applications of artificial intelligence (AI) in healthcare offer the opportunity to improve diagnostic capabilities in obstetrics and gynaecology (O&G), ensuring early detection of pathology, optimal management and improving survival. Consensus on a robust AI healthcare framework is crucial for standardising protocols that promote data privacy and transparency, minimise bias, and ensure patient safety. Here, we describe the study protocol for a systematic review and meta-analysis to evaluate current applications of AI in O&G diagnostics with consideration of reporting standards used and their ethical implications. This protocol is written following the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 checklist. Methods and analysis The study objective is to explore the current application of AI in O&G diagnostics and assess the reporting standards used in these studies. Electronic bibliographic databases MEDLINE, EMBASE and Cochrane will be searched. Study selection, data extraction and subsequent narrative synthesis and meta-analyses will be carried out following the PRISMA-P guidelines. Included papers will be English-language full-text articles from May 2015 to March 2024, which provide original data, as AI has been redefined in recent literature. Papers must use AI as the predictive method, focusing on improving O&G diagnostic outcomes. We will evaluate the reporting standards including the risk of bias, lack of transparency and consider the ethical implications and potential harm to patients. Outcome measures will involve assessing the included studies against gold-standard criteria for robustness of model development (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis, model predictive performance, model risk of bias and applicability (Prediction model Risk Of Bias Assessment Tool and study reporting (Consolidated Standards of Reporting Trials-AI) guidance. Ethics and dissemination Ethical approval is not required for this systematic review. Findings will be shared through peer-reviewed publications.There will be no patient or public involvement in this study. PROSPERO registration number CRD42022357024.

medicine, general & internal
An ethics assessment tool for artificial intelligence implementation in healthcare: CARE-AI

Yilin Ning,Xiaoxuan Liu,Gary S. Collins,Karel G. M. Moons,Melissa McCradden,Daniel Shu Wei Ting,Jasmine Chiat Ling Ong,Benjamin Alan Goldstein,Siegfried K. Wagner,Pearse A. Keane,Eric J. Topol,Nan Liu

DOI: https://doi.org/10.1038/s41591-024-03310-1

IF: 82.9

2024-10-20

Nature Medicine

Abstract:The deployment of artificial intelligence (AI)-powered prediction models in healthcare can lead to ethical concerns about their implementation and upscaling. For example, AI prediction models can hinder clinical decision-making if they advise different diagnoses or treatments by sex and gender or by race and ethnicity without clear justification. Recent guidance (such as the WHO guidance on ethics and governance of AI for health and the Dutch guideline on AI for healthcare) and legislation (such as the European Union AI Act and the White House Executive Order on Safe, Secure, and Trustworthy Development and Use of AI in United States) have outlined important principles for the implementation of AI, including ethical considerations 1,2 . Health systems have responded by establishing governance committees and processes to ensure the safe and equitable implementation of AI tools 3 . However, there is currently no assessment tool that can identify and mitigate ethical issues during the implementation of AI prediction models in healthcare practice, including for public health. The development and validation of AI prediction models has benefited from detailed reporting and risk-of-bias tools, such as TRIPOD+AI 4 and PROBAST (with its forthcoming AI extension) for fairness and bias control and CLAIM 5 for data privacy, security and interpretability of AI imaging studies. However, when planning the implementation of a rigorously developed and well-performing AI prediction model in healthcare practice, existing recommendations and guidance on ethics are sparse and lack operational detail. For example, the DECIDE-AI reporting guideline 6 contains a small number of ethics-related recommendations for early clinical evaluation of AI concerning equity, safety and human-AI interaction, and FUTURE-AI 7 provides recommendations based on six principles (fairness, universality, traceability, usability, robustness and explainability) in model design, development, validation and deployment. A bioethics-centric delivery science toolkit for responsible AI implementation in healthcare is needed 8 .

biochemistry & molecular biology,cell biology,medicine, research & experimental
Re-defining Radiology Quality Assurance (QA) -- Artificial Intelligence (AI)-Based QA by Restricted Investigation of Unequal Scores (AQUARIUS)

Axel Wismueller,Larry Stockmaster,Ali Vosoughi

DOI: https://doi.org/10.48550/arXiv.2205.00629

2022-05-04

Abstract:There is an urgent need for streamlining radiology Quality Assurance (QA) programs to make them better and faster. Here, we present a novel approach, Artificial Intelligence (AI)-Based QUality Assurance by Restricted Investigation of Unequal Scores (AQUARIUS), for re-defining radiology QA, which reduces human effort by up to several orders of magnitude over existing approaches. AQUARIUS typically includes automatic comparison of AI-based image analysis with natural language processing (NLP) on radiology reports. Only the usually small subset of cases with discordant reads is subsequently reviewed by human experts. To demonstrate the clinical applicability of AQUARIUS, we performed a clinical QA study on Intracranial Hemorrhage (ICH) detection in 1936 head CT scans from a large academic hospital. Immediately following image acquisition, scans were automatically analyzed for ICH using a commercially available software (Aidoc, Tel Aviv, Israel). Cases rated positive for ICH by AI (ICH-AI+) were automatically flagged in radiologists' reading worklists, where flagging was randomly switched off with probability 50%. Using AQUARIUS with NLP on final radiology reports and targeted expert neuroradiology review of only 29 discordantly classified cases reduced the human QA effort by 98.5%, where we found a total of six non-reported true ICH+ cases, with radiologists' missed ICH detection rates of 0.52% and 2.5% for flagged and non-flagged cases, respectively. We conclude that AQUARIUS, by combining AI-based image analysis with NLP-based pre-selection of cases for targeted human expert review, can efficiently identify missed findings in radiology studies and significantly expedite radiology QA programs in a hybrid human-machine interoperability approach.

Human-Computer Interaction,Artificial Intelligence
Prospectively investigating the impact of AI onshared decision-making in post kidney transplant care (PRIMA-AI): protocol for a longitudinal qualitative study among patients, their support persons and treating physicians at a tertiary care centre

Zeineb Sassi,Sascha Eickmann,Roland Roller,Bilgin Osmanodja,Aljoscha Burchardt,David Samhammer,Peter Dabrock,Sebastian Möller,Klemens Budde,Anne Herrmann

DOI: https://doi.org/10.1136/bmjopen-2023-081318

IF: 3.006

2024-10-02

BMJ Open

Abstract:Introduction As healthcare is shifting from a paternalistic to a patient-centred approach, medical decision making becomes more collaborative involving patients, their support persons (SPs) and physicians. Implementing shared decision-making (SDM) into clinical practice can be challenging and becomes even more complex with the introduction of artificial intelligence (AI) as a potential actant in the communicative network. Although there is more empirical research on patients' and physicians' perceptions of AI, little is known about the impact of AI on SDM. This study will help to fill this gap. To the best of our knowledge, this is the first systematic empirical investigation to prospectively assess the views of patients, their SPs and physicians on how AI affects SDM in physician–patient communication after kidney transplantation. Using a transdisciplinary approach, this study will explore the role and impact of an AI-decision support system (DSS) designed to assist with medical decision making in the clinical encounter. Methods and analysis This is a plan to roll out a 2 year, longitudinal qualitative interview study in a German kidney transplant centre. Semi-structured interviews with patients, SPs and physicians will be conducted at baseline and in 3-, 6-, 12- and 24-month follow-up. A total of 50 patient–SP dyads and their treating physicians will be recruited at baseline. Assuming a dropout rate of 20% per year, it is anticipated that 30 patient–SP dyads will be included in the last follow-up with the aim of achieving data saturation. Interviews will be audio-recorded and transcribed verbatim. Transcripts will be analysed using framework analysis. Participants will be asked to report on their (a) communication experiences and preferences, (b) views on the influence of the AI-based DSS on the normative foundations of the use of AI in medical decision-making, focusing on agency along with trustworthiness, transparency and responsibility and (c) perceptions of the use of the AI-based DSS, as well as barriers and facilitators to its implementation into routine care. Ethics and dissemination Approval has been granted by the local ethics committee of Charité—Universitätsmedizin Berlin (EA1/177/23 on 08 August 2023). This research will be conducted in accordance with the principles of the Declaration of Helsinki (1996). The study findings will be used to develop communication guidance for physicians on how to introduce and sustainably implement AI-assisted SDM. The study results will also be used to develop lay language patient information on AI-assisted SDM. A broad dissemination strategy will help communicate the results of this research to a variety of target groups, including scientific and non-scientific audiences, to allow for a more informed discourse among different actors from policy, science and society on the role and impact of AI in physician–patient communication.

medicine, general & internal
[Autoxidation products of arachidonic acid modify the arachidonic acid-induced aggregation of human blood platelets].

W. Lösche,E. Michel,U. Till

Abstract:
Simplifying proximal isovelocity surface area as an assessment method of mitral valve area in patients with rheumatic mitral stenosis by fixing aliasing velocity and mitral valve angle.

Alaa Mabrouk Salem Omar,M. A. Abdel-Rahman,Hidekazu Tanaka,O. Rifaie

DOI: https://doi.org/10.1016/j.jsha.2012.11.004

Journal of the Saudi Heart Association

Abstract:
Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Baptiste Vasey,Myura Nagendran,Bruce Campbell,David A Clifton,Gary S Collins,Spiros Denaxas,Alastair K Denniston,Livia Faes,Bart Geerts,Mudathir Ibrahim,Xiaoxuan Liu,Bilal A Mateen,Piyush Mathur,Melissa D McCradden,Lauren Morgan,Johan Ordish,Campbell Rogers,Suchi Saria,Daniel S W Ting,Peter Watkinson,Wim Weber,Peter Wheatstone,Peter McCulloch

DOI: https://doi.org/10.1136/bmj-2022-070904

2022-05-19

The BMJ

Abstract:The prospect of improved clinical outcomes and more efficient health systems has fuelled a rapid rise in the development and evaluation of artificial intelligence (AI) systems over the last decade. Because most AI systems within healthcare are complex interventions designed as clinical decision support systems, rather than autonomous agents, the interactions between the AI systems, their users and the implementation environments are defining components of the AI interventions' overall potential effectiveness. Therefore, bringing AI systems from mathematical performance to clinical utility, needs an adapted, stepwise implementation and evaluation pathway, addressing the complexity of this collaboration between two independent forms of intelligence, beyond measures of effectiveness alone.1 Despite indications that some AI-based algorithms now match the accuracy of human experts within preclinical in silico studies,2 there is little high quality evidence for improved clinician performance or patient outcomes in clinical studies.34 Reasons proposed for this so called AI-chasm5 are lack of...
Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Baptiste Vasey,Myura Nagendran,Bruce Campbell,David A. Clifton,Gary S. Collins,Spiros Denaxas,Alastair K. Denniston,Livia Faes,Bart Geerts,Mudathir Ibrahim,Xiaoxuan Liu,Bilal A. Mateen,Piyush Mathur,Melissa D. McCradden,Lauren Morgan,Johan Ordish,Campbell Rogers,Suchi Saria,Daniel S. W. Ting,Peter Watkinson,Wim Weber,Peter Wheatstone,Peter McCulloch,the DECIDE-AI expert group

DOI: https://doi.org/10.1038/s41591-022-01772-9

IF: 82.9

2022-05-19

Nature Medicine

Abstract:Nature Medicine, Published online: 18 May 2022; doi:10.1038/s41591-022-01772-9 The DECIDE-AI checklist, resulting from a multi-stakeholder group of experts in a Delphi process and following the EQUATOR Network's recommendations, includes key items that should be reported in early-stage clinical studies of AI-based decision support systems, to ensure a responsible and transparent deployment of AI systems in healthcare.

biochemistry & molecular biology,cell biology,medicine, research & experimental
Assessment of bias in scoring of AI-based radiotherapy segmentation and planning studies using modified TRIPOD and PROBAST guidelines as an example

Coen Hurkmans,Jean-Emmanuel Bibault,Enrico Clementel,Jennifer Dhont,Wouter van Elmpt,Georgios Kantidakis,Nicolaus Andratschke

DOI: https://doi.org/10.1016/j.radonc.2024.110196

IF: 6.901

2024-05-01

Radiotherapy and Oncology

Abstract:BACKGROUND AND PURPOSE: Studies investigating the application of Artificial Intelligence (AI) in the field of radiotherapy exhibit substantial variations in terms of quality. The goal of this study was to assess the amount of transparency and bias in scoring articles with a specific focus on AI based segmentation and treatment planning, using modified PROBAST and TRIPOD checklists, in order to provide recommendations for future guideline developers and reviewers.MATERIALS AND METHODS: The TRIPOD and PROBAST checklist items were discussed and modified using a Delphi process. After consensus was reached, 2 groups of 3 co-authors scored 2 articles to evaluate usability and further optimize the adapted checklists. Finally, 10 articles were scored by all co-authors. Fleiss' kappa was calculated to assess the reliability of agreement between observers.RESULTS: Three of the 37 TRIPOD items and 5 of the 32 PROBAST items were deemed irrelevant. General terminology in the items (e.g., multivariable prediction model, predictors) was modified to align with AI-specific terms. After the first scoring round, further improvements of the items were formulated, e.g., by preventing the use of sub-questions or subjective words and adding clarifications on how to score an item. Using the final consensus list to score the 10 articles, only 2 out of the 61 items resulted in a statistically significant kappa of 0.4 or more demonstrating substantial agreement. For 41 items no statistically significant kappa was obtained indicating that the level of agreement among multiple observers is due to chance alone.CONCLUSION: Our study showed low reliability scores with the adapted TRIPOD and PROBAST checklists. Although such checklists have shown great value during development and reporting, this raises concerns about the applicability of such checklists to objectively score scientific articles for AI applications. When developing or revising guidelines, it is essential to consider their applicability to score articles without introducing bias.

oncology,radiology, nuclear medicine & medical imaging
Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence

Gary S Collins,Paula Dhiman,Constanza L Andaur Navarro,Jie Ma,Lotty Hooft,Johannes B Reitsma,Patricia Logullo,Andrew L Beam,Lily Peng,Ben Van Calster,Maarten van Smeden,Richard D Riley,Karel GM Moons

DOI: https://doi.org/10.1136/bmjopen-2020-048008

IF: 3.006

2021-07-01

BMJ Open

Abstract:Introduction The Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis (TRIPOD) statement and the Prediction model Risk Of Bias ASsessment Tool (PROBAST) were both published to improve the reporting and critical appraisal of prediction model studies for diagnosis and prognosis. This paper describes the processes and methods that will be used to develop an extension to the TRIPOD statement (TRIPOD-artificial intelligence, AI) and the PROBAST (PROBAST-AI) tool for prediction model studies that applied machine learning techniques. Methods and analysis TRIPOD-AI and PROBAST-AI will be developed following published guidance from the EQUATOR Network, and will comprise five stages. Stage 1 will comprise two systematic reviews (across all medical fields and specifically in oncology) to examine the quality of reporting in published machine-learning-based prediction model studies. In stage 2, we will consult a diverse group of key stakeholders using a Delphi process to identify items to be considered for inclusion in TRIPOD-AI and PROBAST-AI. Stage 3 will be virtual consensus meetings to consolidate and prioritise key items to be included in TRIPOD-AI and PROBAST-AI. Stage 4 will involve developing the TRIPOD-AI checklist and the PROBAST-AI tool, and writing the accompanying explanation and elaboration papers. In the final stage, stage 5, we will disseminate TRIPOD-AI and PROBAST-AI via journals, conferences, blogs, websites (including TRIPOD, PROBAST and EQUATOR Network) and social media. TRIPOD-AI will provide researchers working on prediction model studies based on machine learning with a reporting guideline that can help them report key details that readers need to evaluate the study quality and interpret its findings, potentially reducing research waste. We anticipate PROBAST-AI will help researchers, clinicians, systematic reviewers and policymakers critically appraise the design, conduct and analysis of machine learning based prediction model studies, with a robust standardised tool for bias evaluation. Ethics and dissemination Ethical approval has been granted by the Central University Research Ethics Committee, University of Oxford on 10-December-2020 (R73034/RE001). Findings from this study will be disseminated through peer-review publications. PROSPERO registration number CRD42019140361 and CRD42019161764.

medicine, general & internal
[Children's preventive dentistry (4). Dental health education in Vienna kindergarten].

I. Gal

1986-05-01

Abstract:

ModelEd, TestEd, TrustEd

APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support

[Eosinophilic globules in the Spitz nevus].

[Risk on Bias Assessment: (6) A Revised Tool for the Quality Assessment on Diagnostic Accuracy Studies (QUADAS-2)].

Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platforms

Stakeholder perspectives towards diagnostic artificial intelligence: a co-produced qualitative evidence synthesis

Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review

A Comprehensive, Valid, and Reliable Tool to Assess the Degree of Responsibility of Digital Health Solutions That Operate With or Without Artificial Intelligence: 3-Phase Mixed Methods Study

Detecting Algorithmic Errors and Patient Harms for AI-Enabled Medical Devices in Randomized Controlled Trials: Protocol for a Systematic Review

Use of artificial intelligence in obstetric and gynaecological diagnostics: a protocol for a systematic review and meta-analysis

An ethics assessment tool for artificial intelligence implementation in healthcare: CARE-AI

Re-defining Radiology Quality Assurance (QA) -- Artificial Intelligence (AI)-Based QA by Restricted Investigation of Unequal Scores (AQUARIUS)

Prospectively investigating the impact of AI onshared decision-making in post kidney transplant care (PRIMA-AI): protocol for a longitudinal qualitative study among patients, their support persons and treating physicians at a tertiary care centre

[Autoxidation products of arachidonic acid modify the arachidonic acid-induced aggregation of human blood platelets].

Simplifying proximal isovelocity surface area as an assessment method of mitral valve area in patients with rheumatic mitral stenosis by fixing aliasing velocity and mitral valve angle.

Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Assessment of bias in scoring of AI-based radiotherapy segmentation and planning studies using modified TRIPOD and PROBAST guidelines as an example

Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence

[Children's preventive dentistry (4). Dental health education in Vienna kindergarten].