Abstract:Background: Though there has been considerable effort to implement machine learning (ML) methods for health care, clinical implementation has lagged. Incorporating explainable machine learning (XML) methods through the development of a decision support tool using a design thinking approach is expected to lead to greater uptake of such tools. Objective: This work aimed to explore how constant engagement of clinician end users can address the lack of adoption of ML tools in clinical contexts due to their lack of transparency and address challenges related to presenting explainability in a decision support interface. Methods: We used a design thinking approach augmented with additional theoretical frameworks to provide more robust approaches to different phases of design. In particular, in the problem definition phase, we incorporated the nonadoption, abandonment, scale-up, spread, and sustainability of technology in health care (NASSS) framework to assess these aspects in a health care network. This process helped focus on the development of a prognostic tool that predicted the likelihood of admission to an intensive care ward based on disease severity in chest x-ray images. In the ideate, prototype, and test phases, we incorporated a metric framework to assess physician trust in artificial intelligence (AI) tools. This allowed us to compare physicians' assessments of the domain representation, action ability, and consistency of the tool. Results: Physicians found the design of the prototype elegant, and domain appropriate representation of data was displayed in the tool. They appreciated the simplified explainability overlay, which only displayed the most predictive patches that cumulatively explained 90% of the final admission risk score. Finally, in terms of consistency, physicians unanimously appreciated the capacity to compare multiple x-ray images in the same view. They also appreciated the ability to toggle the explainability overlay so that both options made it easier for them to assess how consistently the tool was identifying elements of the x-ray image they felt would contribute to overall disease severity. Conclusions: The adopted approach is situated in an evolving space concerned with incorporating XML or AI technologies into health care software. We addressed the alignment of AI as it relates to clinician trust, describing an approach to wire framing and prototyping, which incorporates the use of a theoretical framework for trust in the design process itself. Moreover, we proposed that alignment of AI is dependent upon integration of end users throughout the larger design process. Our work shows the importance and value of engaging end users prior to tool development. We believe that the described approach is a unique and valuable contribution that outlines a direction for ML experts, user experience designers, and clinician end users on how to collaborate in the creation of trustworthy and usable XML-based clinical decision support tools.

User-Centric Explainability in Healthcare: A Knowledge-Level Perspective of Informed Machine Learning

Leveraging Clinical Context for User-Centered Explainability: A Diabetes Use Case

The Explanation Necessity for Healthcare AI

Meaningful Explanation Effect on User’s Trust in an AI Medical System: Designing Explanations for Non-Expert Users

An Ontology-Enabled Approach For User-Centered and Knowledge-Enabled Explanations of AI Systems

Explainability as fig leaf? An exploration of experts’ ethical expectations towards machine learning in psychiatry

Explainable, trustworthy, and ethical machine learning for healthcare: A survey

What do we need to build explainable AI systems for the medical domain?

Rethinking explainability: toward a postphenomenology of black-box artificial intelligence in medicine

Explanation Ontology: A Model of Explanations for User-Centered AI

Foundations of Explainable Knowledge-Enabled Systems

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

A novel framework for artificial intelligence explainability via the Technology Acceptance Model and Rapid Estimate of Adult Literacy in Medicine using machine learning

The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies

Explainable Artificial Intelligence for Predictive Modeling in Healthcare

Interpretable and explainable machine learning: A methods‐centric overview with concrete examples

Lessons Learned from EXMOS User Studies: A Technical Report Summarizing Key Takeaways from User Studies Conducted to Evaluate The EXMOS Platform

Integrating Explainable Machine Learning in Clinical Decision Support Systems: Study Involving a Modified Design Thinking Approach

Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classification

Explainable Artificial Intelligence for Human Decision-Support System in Medical Domain

Rethinking Explainability as a Dialogue: A Practitioner's Perspective