Conversational Medical AI: Ready for Practice

Antoine Lizée,Pierre-Auguste Beaucoté,James Whitbeck,Marion Doumeingts,Anaël Beaugnon,Isabelle Feldhaus

2024-11-20

Abstract:The shortage of doctors is creating a critical squeeze in access to medical expertise. While conversational Artificial Intelligence (AI) holds promise in addressing this problem, its safe deployment in patient-facing roles remains largely unexplored in real-world medical settings. We present the first large-scale evaluation of a physician-supervised LLM-based conversational agent in a real-world medical setting. Our agent, Mo, was integrated into an existing medical advice chat service. Over a three-week period, we conducted a randomized controlled experiment with 926 cases to evaluate patient experience and satisfaction. Among these, Mo handled 298 complete patient interactions, for which we report physician-assessed measures of safety and medical accuracy. Patients reported higher clarity of information (3.73 vs 3.62 out of 4, p < 0.05) and overall satisfaction (4.58 vs 4.42 out of 5, p < 0.05) with AI-assisted conversations compared to standard care, while showing equivalent levels of trust and perceived empathy. The high opt-in rate (81% among respondents) exceeded previous benchmarks for AI acceptance in healthcare. Physician oversight ensured safety, with 95% of conversations rated as "good" or "excellent" by general practitioners experienced in operating a medical advice chat service. Our findings demonstrate that carefully implemented AI medical assistants can enhance patient experience while maintaining safety standards through physician supervision. This work provides empirical evidence for the feasibility of AI deployment in healthcare communication and insights into the requirements for successful integration into existing healthcare services.

Artificial Intelligence,Computers and Society,Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use conversational artificial intelligence (AI) technology to improve the accessibility and quality of medical services in the context of a shortage of medical resources. Specifically, the research focuses on the effectiveness and safety of deploying doctor - supervised large - language - model (LLM) - based conversational AI agents (called Mo) in real - world medical environments. By evaluating patient experience and satisfaction, as well as doctors' evaluations of safety and medical accuracy, the paper aims to explore whether this technology can safely and effectively assist doctors in actual medical scenarios, improve patients' medical experiences, and maintain medical service quality standards.