Technology-Supported Self-Triage Decision Making: A Mixed-Methods Study

Marvin Kopka,Sonja Mei Wang,Samira Kunz,Christine Schmid,Markus A. Feufel

DOI: https://doi.org/10.1101/2024.09.12.24313558

2024-09-13

Abstract:Symptom-Assessment Application (SAAs) and Large Language Models (LLMs) are increasingly used by laypeople to navigate care options. Although humans ultimately make a final decision when using these systems, previous research has typically examined the performance of humans and SAAs/LLMs separately. Thus, it is unclear how decision-making unfolds in such hybrid human-technology teams and if SAAs/LLMs can improve laypeople's decisions. To address this gap, we conducted a convergent parallel mixed-methods study with semi-structured interviews and a randomized controlled trial. Our interview data revealed that in human-technology teams, decision-making is influenced by factors before, during, and after interaction. Users tend to rely on technology for information gathering and analysis but remain responsible for information integration and the final decision. Based on these results, we developed a model for technology-assisted self-triage decision-making. Our quantitative results indicate that when using a high-performing SAA, laypeople's decision accuracy improved from 53.2% to 64.5% (OR = 2.52, p < .001). In contrast, decision accuracy remained unchanged when using a LLM (54.8% before vs. 54.2% after usage, p = .79). These findings highlight the importance of studying SAAs/LLMs with humans in the loop, as opposed to analyzing them in isolation.

Health Informatics

What problem does this paper attempt to address?

The paper aims to address the following key issues:

1. **Understanding the interaction between humans and technological systems (such as Symptom Assessment Applications (SAAs) and Large Language Models (LLMs)) in self-triage decision-making**: The study explores how humans integrate these technologies into their decision-making processes and analyzes the specific influencing factors of this human-machine collaboration.

2. **Evaluating whether SAAs and LLMs can improve the accuracy of self-triage decisions made by non-professionals**: Through empirical research, particularly Randomized Controlled Trials (RCTs), the study verifies whether the use of high-performance SAAs or LLMs can significantly enhance the decision accuracy of ordinary people.

Specifically, the paper points out that although there is extensive research on the performance of SAAs or LLMs individually, there is a lack of studies on the system performance in a "human-in-the-loop" context. This paper attempts to fill this gap by exploring the following two main research questions through a mixed-methods approach (including semi-structured interviews and randomized controlled trials):
- How do human-machine teams make decisions?
- Does the accuracy of self-triage decisions made by non-professionals improve when using high-performance SAAs or LLMs?