Exploring the Requirements of Clinicians for Explainable AI Decision Support Systems in Intensive Care

Jeffrey N. Clark,Matthew Wragg,Emily Nielsen,Miquel Perello-Nieto,Nawid Keshtmand,Michael Ambler,Shiv Sharma,Christopher P. Bourdeaux,Amberly Brigden,Raul Santos-Rodriguez

2024-11-19

Abstract:There is a growing need to understand how digital systems can support clinical decision-making, particularly as artificial intelligence (AI) models become increasingly complex and less human-interpretable. This complexity raises concerns about trustworthiness, impacting safe and effective adoption of such technologies. Improved understanding of decision-making processes and requirements for explanations coming from decision support tools is a vital component in providing effective explainable solutions. This is particularly relevant in the data-intensive, fast-paced environments of intensive care units (ICUs). To explore these issues, group interviews were conducted with seven ICU clinicians, representing various roles and experience levels. Thematic analysis revealed three core themes: (T1) ICU decision-making relies on a wide range of factors, (T2) the complexity of patient state is challenging for shared decision-making, and (T3) requirements and capabilities of AI decision support systems. We include design recommendations from clinical input, providing insights to inform future AI systems for intensive care.

Human-Computer Interaction,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How to design and implement an interpretable artificial intelligence (xAI) decision - support system to meet the needs of intensive care unit (ICU) clinicians. Specifically, the paper aims to understand the decision - making processes of ICU clinicians when currently using data and decision - support systems, and clarify their specific requirements for interpretable AI systems, thereby increasing the acceptance and effectiveness of these systems in actual clinical work.

### Main research questions:
1. **RQ1**: How do clinicians currently use data and decision - support systems to make and interpret decisions in intensive care?
2. **RQ2**: What are the specific requirements of clinicians for the design and implementation of interpretable AI decision - support systems in intensive care?

### Research background:
As artificial intelligence (AI) models become more and more complex and difficult to interpret, their applications in the medical field face challenges in terms of trust and security. Especially in the intensive care unit (ICU), due to the data - intensive environment and fast - paced nature, the interpretability of AI systems is particularly important. To ensure the effectiveness and safety of AI systems, it is necessary to understand the needs of clinicians for these systems and ensure that they can be integrated into the existing work - flow.

### Research methods:
Researchers conducted semi - structured interviews with seven ICU clinicians from different roles and experience levels through two 45 - minute workshops. Through thematic analysis of the interview transcripts, the researchers identified three core themes:

1. **ICU decision - making depends on a wide range of factors**:
- Patient course: including treatment plans, patient trajectories, historical and recent data.
- Patient physiological data: such as organ systems, diagnosis - specific data, baseline changes and trends.
- Importance of bedside information: forming a complete patient profile (e.g., behavioral and visual factors).
- Operational factors: such as bed and staff availability, hospital flow and resource constraints.

2. **The complexity of patient status poses challenges to shared decision - making**:
- Collaborative decision - making: checking decisions with colleagues, prompting investigations and family participation.
- Patient handover between clinicians: handover efficiency and information transfer framework.

3. **Requirements and capabilities of AI decision - support systems**:
- Framework and presentation of decision - making suggestions: providing evidence to support suggestions, promoting acceptance and trust.
- Complexity and detail of explanations: hierarchical systems, minimizing cognitive load and the number of systems.
- Impact of xAI suggestions on decision - making: avoiding bias or ambiguous judgments, prompting new ideas and highlighting missing information.

### Conclusions:
Through in - depth exploration of these themes, researchers have provided valuable insights for the future development of AI systems that are more in line with the needs of the ICU. This not only helps to improve the interaction ability between clinicians and AI systems, but also promotes the adoption of these systems, and ultimately improves the quality of patient care.

### Formula examples:
Although this article does not involve specific mathematical formulas, when discussing AI interpretability, some common machine - learning interpretation methods may be used, such as SHAP values (Shapley Additive Explanations) and LIME (Local Interpretable Model - agnostic Explanations). These methods can help explain the prediction results of AI models.

For example, SHAP values can be calculated by the following formula:
\[ \phi_i(v) = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!(n - |S| - 1)!}{n!} [v(S \cup \{i\}) - v(S)] \]

where \( \phi_i(v) \) represents the contribution value of feature \( i \), \( v(S) \) represents the value function of set \( S \), and \( n \) is the total number of features.

In this way, researchers can better understand the decision - making process of AI models, thereby improving their interpretability and credibility.