Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes

Qian Yang,Aaron Steinfeld,John Zimmerman

DOI: https://doi.org/10.1145/3290605.3300468

2019-04-21

Abstract:Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of "Unremarkable Computing", that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.

Human-Computer Interaction,Artificial Intelligence,Computers and Society,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the failure of clinical decision - support tools (DSTs) in the transition from the laboratory environment to actual clinical applications. Although these tools perform well in the laboratory environment, they almost all fail in actual clinical practice. Empirical research has found that the main reason for the failure of these tools is the lack of fit with the clinical workflow. Specifically, the paper points out:

1. **Lack of consideration for the clinical workflow**: Most existing DSTs are designed on the assumption that individual clinicians will actively use these systems when they need help, but in reality, clinicians rarely leave their daily workflow to find or use these tools.

2. **Ignoring the collaborative nature of clinical work**: Clinical decision - making is usually a teamwork process, but existing DSTs often overlook this point and fail to effectively integrate into the team's workflow.

3. **Cultural and technological barriers**: The medical culture is highly stratified, and there is almost no intersection between senior physicians and intermediate - level clinicians who use computers. In addition, many clinical decisions are not made in front of a computer, which further limits the application of DSTs.

To solve these problems, the paper proposes a brand - new DST design method, namely **Unremarkable AI** (Unremarkable AI). The core idea of this design method is to embed the prediction results of machine learning into the daily work of clinicians in an unobtrusive way, thereby increasing the acceptance and effectiveness of DSTs in actual clinical practice. Specifically, the paper designs a system that automatically generates conference slides, and these slides embed the prediction results of machine learning in the corner, allowing clinicians to naturally access this information during decision - making meetings without interrupting their normal workflow.

In this way, the paper aims to explore how to more effectively integrate DSTs into clinical practice and improve their success rate in practical applications.