Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned

Sally L Baxter,Christopher A Longhurst,Marlene Millen,Amy M Sitapati,Ming Tai-Seale

DOI: https://doi.org/10.1093/jamiaopen/ooae028

2024-04-10

JAMIA Open

Abstract:Background: Electronic health record (EHR)-based patient messages can contribute to burnout. Messages with a negative tone are particularly challenging to address. In this perspective, we describe our initial evaluation of large language model (LLM)-generated responses to negative EHR patient messages and contend that using LLMs to generate initial drafts may be feasible, although refinement will be needed. Methods: A retrospective sample (n = 50) of negative patient messages was extracted from a health system EHR, de-identified, and inputted into an LLM (ChatGPT). Qualitative analyses were conducted to compare LLM responses to actual care team responses. Results: Some LLM-generated draft responses varied from human responses in relational connection, informational content, and recommendations for next steps. Occasionally, the LLM draft responses could have potentially escalated emotionally charged conversations. Conclusion: Further work is needed to optimize the use of LLMs for responding to negative patient messages in the EHR.