Abstract:The field of radiology imaging has experienced a remarkable increase in using of deep learning (DL) algorithms to support diagnostic and treatment decisions. This rise has led to the development of Explainable AI (XAI) system to improve the transparency and trust of complex DL methods. However, XAI systems face challenges in gaining acceptance within the healthcare sector, mainly due to technical hurdles in utilizing these systems in practice and the lack of human-centered evaluation/validation. In this study, we focus on visual XAI systems applied to DL-enabled diagnostic system in chest radiography. In particular, we conduct a user study to evaluate two prominent visual XAI techniques from the human perspective. To this end, we created two clinical scenarios for diagnosing pneumonia and COVID-19 using DL techniques applied to chest X-ray and CT scans. The achieved accuracy rates were 90% for pneumonia and 98% for COVID-19. Subsequently, we employed two well-known XAI methods, Grad-CAM (Gradient-weighted Class Activation Mapping) and LIME (Local Interpretable Model-agnostic Explanations), to generate visual explanations elucidating the AI decision-making process. The visual explainability results were shared through a user study, undergoing evaluation by medical professionals in terms of clinical relevance, coherency, and user trust. In general, participants expressed a positive perception of the use of XAI systems in chest radiography. However, there was a noticeable lack of awareness regarding their value and practical aspects. Regarding preferences, Grad-CAM showed superior performance over LIME in terms of coherency and trust, although concerns were raised about its clinical usability. Our findings highlight key user-driven explainability requirements, emphasizing the importance of multi-modal explainability and the necessity to increase awareness of XAI systems among medical practitioners. Inclusive design was also identified as a crucial need to ensure better alignment of these systems with user needs.

A Vision on User-Centered Implementation and Evaluation of Explainable AI for Predicting Hospital-Onset Bacteremia

A Comparative Approach to Explainable Artificial Intelligence Methods in Application to High-Dimensional Electronic Health Records: Examining the Usability of XAI

Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare

Explainable Artificial Intelligence for Predictive Modeling in Healthcare

Explainable Artificial Intelligence (XAI) in Biomedicine: Making AI Decisions Trustworthy for Physicians and Patients

A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?

Evaluation of Popular XAI Applied to Clinical Prediction Models: Can They be Trusted?

A Brief Review of Explainable Artificial Intelligence in Healthcare

User Trust and Understanding of Explainable AI: Exploring Algorithm Visualisations and User Biases

What Do End-Users Really Want? Investigation of Human-Centered XAI for Mobile Health Apps

Explainable Artificial Intelligence for Medical Applications: A Review

A Survey on Medical Explainable AI (XAI): Recent Progress, Explainability Approach, Human Interaction and Scoring System

Study on the Helpfulness of Explainable Artificial Intelligence

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Evaluating Explanations from AI Algorithms for Clinical Decision-Making: A Social Science-based Approach

Co-design of human-centered, explainable AI for clinical decision support

Evaluating Explainable Artificial Intelligence (XAI) techniques in chest radiology imaging through a human-centered Lens

Designing Theory-Driven User-Centric Explainable AI

Explainable AI for clinical and remote health applications: a survey on tabular and time series data

The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies