Daily briefing: Testers say Google AI has a better bedside manner than human doctors

Flora Graham

DOI: https://doi.org/10.1038/d41586-024-00139-z

IF: 64.8

2024-01-16

Nature

Abstract:A Google AI system outperformed human physicians in diagnosing certain medical conditions ranked higher on empathy. Plus, a mysterious fossil skull could belong to new tyrannosaur species and why great apes are dying from our colds.

multidisciplinary sciences

What problem does this paper attempt to address?

Google AI has better bedside manner than human doctors — and makes better diagnoses

Mariana Lenharo

DOI: https://doi.org/10.1038/d41586-024-00099-4

IF: 64.8

2024-01-13

Nature

Abstract:Researchers say their artificial-intelligence system could help to democratize medicine. Researchers say their artificial-intelligence system could help to democratize medicine.

multidisciplinary sciences
[Rare disease in the age of artificial intelligence]

Carlo Alfredo Clerici,Saba Chopard,Giuseppe Levi

DOI: https://doi.org/10.1701/4197.41839

Abstract:Introduction: The text examines the impact of artificial intelligence (AI) in the context of rare diseases, exploring how patients turn to AI resources for health information, especially in situations where doctor-patient communication is limited. The article features the case of a doctor specializing in clinical psychology and psychotherapy, diagnosed with thymoma and Good's syndrome, who uses AI resources during his illness. Methods: The capabilities of five chatbots based on Large Language Models (LLMs), such as GPT-3.5, GPT-4, Bing Chat, Google Bard, and Anthropic Claude are explored. The AIs were queried on various aspects of the disease, from pre-diagnosis and diagnosis to therapeutic, psychological, and caregiver management issues. The responses were evaluated by five experts based on criteria such as: accuracy, relevance, coherence, clarity, practical utility, ethical considerations, empathy, and capacity to respond to questions and concerns. Results: The results indicate consistency in the evaluators' assessments, with generally high scores across all dimensions. Particularly, systems like Bard and GPT-4 received high ratings in terms of information accuracy and the ability to respond to questions and concerns. Bing and Claude were appreciated for their empathy and tone. Overall, the AI systems' responses were considered appropriate, respectful of ethics and privacy, and useful in the clinical context. Discussion: The article emphasizes the importance of understanding the reliability and precision of responses provided by AI systems in the clinical field. Although these systems offer high-quality responses, there is significant variability in their performance. Healthcare professionals must be aware of these differences and use such tools cautiously. AI can provide support in some aspects of care but cannot replace genuine human empathy and understanding. Integrating AI into clinical practice presents potential but also challenges, particularly the possibility of providing incorrect information. Conclusions: The AI systems demonstrate the ability to provide useful advice on clinical and psychological issues, but their use requires caution. It is crucial to distinguish the benefits of AI for patients from the challenges it presents for healthcare professionals. As AI technology continues to evolve, it is essential that its integration into the clinical field is accompanied by continuous research and evaluations, to ensure safe and effective use in the healthcare sector.
Doctor AI? A pilot study examining responses of artificial intelligence to common questions asked by geriatric patients

Ian Moore,Christopher Magnante,Ellie Embry,Jennifer Mathis,Scott Mooney,Shereen Haj-Hassan,Maria Cottingham,Prasad R. Padala

DOI: https://doi.org/10.3389/frai.2024.1438012

2024-07-25

Frontiers in Artificial Intelligence

Abstract:Introduction AI technologies have the potential to transform patient care. AI has been used to aid in differential diagnosis and treatment planning for psychiatric disorders, administer therapeutic protocols, assist with interpretation of cognitive testing, and patient treatment planning. Despite advancements, AI has notable limitations and remains understudied and further research on its strengths and limitations in patient care is required. This study explored the responses of AI (Chat-GPT 3.5) and trained clinicians to commonly asked patient questions. Methods Three clinicians and AI provided responses to five dementia/geriatric healthcare-related questions. Responses were analyzed by a fourth, blinded clinician for clarity, accuracy, relevance, depth, and ease of understanding and to determine which response was AI generated. Results AI responses were rated highest in ease of understanding and depth across all responses and tied for first for clarity, accuracy, and relevance. The rating for AI generated responses was 4.6/5 (SD = 0.26); the clinician s' responses were 4.3 (SD = 0.67), 4.2 (SD = 0.52), and 3.9 (SD = 0.59), respectively. The AI generated answers were identified in 4/5 instances. Conclusions AI responses were rated more highly and consistently on each question individually and overall than clinician answers demonstrating that AI could produce good responses to potential patient questions. However, AI responses were easily distinguishable from those of clinicians. Although AI has the potential to positively impact healthcare, concerns are raised regarding difficulties discerning AI from human generated material, the increased potential for proliferation of misinformation, data security concerns, and more.
Following Medical Advice of an AI or a Human Doctor? Experimental Evidence Based on Clinician-Patient Communication Pathway Model

Shuoshuo Li,Meng Chen,Piper Liping Liu,Jian Xu

DOI: https://doi.org/10.1080/10410236.2024.2423114

IF: 3.501

2024-11-06

Health Communication

Abstract:Medical large language models are being introduced to the public in collaboration with governments, medical institutions, and artificial intelligence (AI) researchers. However, a crucial question remains: Will patients follow the medical advice provided by AI doctors? The lack of user research makes it difficult to provide definitive answers. Based on the clinician-patient communication pathway model, this study conducted a factorial experiment with a 2 (medical provider, AI vs. human) × 2 (information support, low vs. high) × 2 (response latency, slow vs. fast) between-subjects design ( n = 535). The results showed that participants exhibited significantly lower adherence to AI doctors' advice than to human doctors. In addition, the interaction effect suggested that, under the slow-response latency condition, subjects perceived greater health benefits and patient-centeredness from human doctors, while the opposite was observed for AI doctors.

health policy & services,communication
Could artificial intelligence make human doctors obsolete? A survey based on public attitudes

Y Y Chen,Y Li,C J Li

DOI: https://doi.org/10.1093/eurpub/ckaa166.040

2020-09-01

European Journal of Public Health

Abstract:Abstract Background With the outbreak of pandemics such as SARS, MERS and COVID-19 etc., making full use of artificial intelligence (AI) become the focus of social discussion. Some experts think AI can reduce infection probability of medical staffs and improve medical efficiency, while opponents believe AI will increase unemployment and medical ethical crises. We try to answer the question: From public perspective, could AI replace human doctors? Methods According to occupational status and biological gender, we select four groups as respondents: on-job and retired men, on-job and retired women. We investigate their attitudes towards AI application in medical field. After collecting 400 anonymous questionnaires, we use sample statistics, ANOVA and cluster method (CFSFDP) to find out cognitive similarities and differences between different groups. Results Reliability (Cronbach's alpha coefficient) and validity (KMO test) of questionnaires are 0.740 and 0.875 respectively. To the four groups, there are significant (p < 0.01) differences in answers of 7 questions but non-significant differences in the remaining 3 questions. By CFSFDP, questionnaires are automatically grouped into two clusters. Largest group in two clusters are on-job women and on-job men, respectively. Although 82.5% of respondents pay attention to news about AI, the overall average score for “public attitude to AI application in medical field” is 2.828 (full-score is 5.000), below neutral-score (3.000). Conclusions To AI application in medical field, cognitive differences among different groups are in existence, especially between on-job men and women. But in general, the public held a negative attitude towards the AI application in medical field, and trust human doctors more. Key messages According to the survey, it seems that AI is lesser probability to replace human doctors in recent years. It is also necessary to consider the public attitudes when promoting medical AI.

public, environmental & occupational health
Artificial intelligence powers digital medicine

Alexander L. Fogel,Joseph C. Kvedar

DOI: https://doi.org/10.1038/s41746-017-0012-2

IF: 15.2

2018-03-14

npj Digital Medicine

Abstract:Artificial intelligence (AI) has recently surpassed human performance in several domains, and there is great hope that in healthcare, AI may allow for better prevention, detection, diagnosis, and treatment of disease. While many fear that AI will disrupt jobs and the physician–patient relationship, we believe that AI can eliminate many repetitive tasks to clear the way for human-to-human bonding and the application of emotional intelligence and judgment. We review several recent studies of AI applications in healthcare that provide a view of a future where healthcare delivery is a more unified, human experience.

health care sciences & services,medical informatics
A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis

Salman Razzaki,Adam Baker,Yura Perov,Katherine Middleton,Janie Baxter,Daniel Mullarkey,Davinder Sangar,Michael Taliercio,Mobasher Butt,Azeem Majeed,Arnold DoRosario,Megan Mahoney,Saurabh Johri

DOI: https://doi.org/10.48550/arXiv.1806.10698

IF: 14.4

2018-06-27

Artificial Intelligence

Abstract:Online symptom checkers have significant potential to improve patient care, however their reliability and accuracy remain variable. We hypothesised that an artificial intelligence (AI) powered triage and diagnostic system would compare favourably with human doctors with respect to triage and diagnostic accuracy. We performed a prospective validation study of the accuracy and safety of an AI powered triage and diagnostic system. Identical cases were evaluated by both an AI system and human doctors. Differential diagnoses and triage outcomes were evaluated by an independent judge, who was blinded from knowing the source (AI system or human doctor) of the outcomes. Independently of these cases, vignettes from publicly available resources were also assessed to provide a benchmark to previous studies and the diagnostic component of the MRCGP exam. Overall we found that the Babylon AI powered Triage and Diagnostic System was able to identify the condition modelled by a clinical vignette with accuracy comparable to human doctors (in terms of precision and recall). In addition, we found that the triage advice recommended by the AI System was, on average, safer than that of human doctors, when compared to the ranges of acceptable triage provided by independent expert judges, with only a minimal reduction in appropriateness.
Behavioral and neural evidence on consumer responses to human doctors and medical artificial intelligence

Jin Ho Yun,Eun‐Ju Lee,Dong Hyun Kim

DOI: https://doi.org/10.1002/mar.21445

2021-01-08

Psychology and Marketing

Abstract:<p>Will consumers accept artificial intelligence (AI) as a medical care provider? On the basis of evolution theory, we investigate the implicit psychological mechanisms that underlie consumers' interactions with medical AI and a human doctor. In a behavioral investigation (Study 1), consumers expressed a positive intention to use medical AI's healthcare services when it used personalized rather than mechanical conversation. However, neural investigation (Study 2) using functional magnetic resonance imaging revealed that some consumers' implicit attitudes toward medical AI differed from their expressed behavioral intentions. The brain areas linked with implicitly apathetic emotions were activated even when medical AI used a personalized conversation, whereas consumers' brains were activated in areas associated with prosociality when they interacted with a human doctor who used a personalized conversation. On the basis of our neural evidence, consumers perceive an identical personalized conversation differently when it is offered by a medical AI versus a human doctor. These findings have implications for the area of human–AI interactions and medical decision‐making and suggest that replacing human doctors with medical AI is still an unrealistic proposition.</p>

business,psychology, applied
Do patients prefer a human doctor, artificial intelligence, or a blend, and is this preference dependent on medical discipline? Empirical evidence and implications for medical practice

René Riedl,Svea A Hogeterp,Martin Reuter

DOI: https://doi.org/10.3389/fpsyg.2024.1422177

2024-08-12

Abstract:Today the doctor-patient relationship typically takes place in a face-to-face setting. However, with the advent of artificial intelligence (AI) systems, two further interaction scenarios are possible: an AI system supports the doctor's decision regarding diagnosis and/or treatment while interacting with the patient, or an AI system could even substitute the doctor and hence a patient interacts with a chatbot (i.e., a machine) alone. Against this background, we report on an online experiment in which we analyzed data from N = 1,183 people. The data was collected in German-speaking countries (Germany, Austria, Switzerland). The participants were asked to imagine they had been suffering from medical conditions of unknown origin for some time and that they were therefore visiting a health center to seek advice from a doctor. We developed descriptions of patient-doctor interactions (referred to as vignettes), thereby manipulating the patient's interaction partner: (i) human doctor, (ii) human doctor with an AI system, and (iii) an AI system only (i.e., chatbot). Furthermore, we manipulated medical discipline: (i) cardiology, (ii) orthopedics, (iii) dermatology, and (iv) psychiatry. Based on this 3 × 4 experimental within-subjects design, our results indicate that people prefer a human doctor, followed by a human doctor with an AI system, and an AI system alone came in last place. Specifically, based on these 12 hypothetical interaction situations, we found a significant main effect of a patient's interaction partner on trust, distrust, perceived privacy invasion, information disclosure, treatment adherence, and satisfaction. Moreover, perceptions of trust, distrust, and privacy invasion predicted information disclosure, treatment adherence, and satisfaction as a function of interaction partner and medical discipline. We found that the situation in psychiatry is different from the other three disciplines. Specifically, the six outcome variables differed strongly between psychiatry and the three other disciplines in the "human doctor with an AI system" condition, while this effect was not that strong in the other conditions (human doctor, chatbot). These findings have important implications for the use of AI in medical care and in the interaction between patients and their doctors.
When an AI Doctor Gets Personal: The Effects of Social and Medical Individuation in Encounters With Human and AI Doctors

Cheng Chen,Mengqi Liao,Joseph B. Walther,S. Shyam Sundar

DOI: https://doi.org/10.1177/00936502241263482

IF: 6.3

2024-08-05

Communication Research

Abstract:Communication Research, Ahead of Print. How do we know when someone knows us? Does it matter whether the knower is a human or a machine? Following the theory of interpersonal knowledge, a between-subjects experiment investigated whether a doctor's incorporation of individualized knowledge about a patient's social or medical history enhances doctor-patient relationships in online conversations. Patients in this study conversed with either a human doctor, an AI doctor, or an AI-assisted human doctor. Following previous research, additional factors such as perceptions of effort, relational closeness, privacy intrusiveness, and the provision of privacy control were assessed. Results showed that an AI doctor enhanced patient satisfaction when it employed social individuation messages, which triggered perceptions of increased effort, but only when patients could activate privacy control. Perception of relational closeness with a human doctor and an AI-assisted human doctor did not seem to require social individuation and privacy control. The study concludes with implications for the theory of interpersonal knowledge and AI-mediated communication research, as well as practical implications for improving chatbot medical systems.

communication
When Doctors Meet with AlphaGo: Potential Application of Machine Learning to Clinical Medicine.

Zhongheng Zhang

DOI: https://doi.org/10.21037/atm.2016.03.25

IF: 3.616

2016-01-01

Annals of Translational Medicine

Abstract:March is a reluctant beginning of spring. After suddenly getting warmer, the weather has turned cold again. Yesterday, the friend circle of Wechat was full of a breaking news that the Google’s artificial intelligence AlphaGo beat human Go champion Lee Sedol. Or at least it was a temporary lead because there will be several rounds in following days. The news was a headline because it was related to the debate on whether artificial intelligence could defeat human brain. Many cultural celebrities posted their comments on the news. Shen Lei, a writer who is famous for his interpretation on Jin Yong’s Kung Fu novels, stated that: “ The last warrior falls. ” Tang Fenng also posted on his Wechat that: “ All calculations and computations can be left for machines from now on. What we humans can do are: burning incense, sipping tea, Washing inkstone, playing the Chinese zither…. ” In other words, all works with rules can be completed by machines, and humans have to retreat to their emotional domain. Why is it the Go, instead of other chesses such as Modern Ludo and animal checker, or other intellectual activities? That is because the Go represent the most complicated intellectual activity. The chessboard of Go is made up of 361 cross points, which in turn can generate numerous chess compositions. There are around 10 170 possible board configurations in the average 150-move game (1)! After all, this is much challenging than medical decision making. Two decades ago, the IBM supercomputer Deep Blue defeated the world chess champion Garry Kasparov (2). However, the chess is slightly inferior than Go in its complexity. Thus the AlphaGo’s victory has more far-reaching significance.
"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making

Carrie J. Cai,Samantha Winter,David Steiner,Lauren Wilcox,Michael Terry

DOI: https://doi.org/10.1145/3359206

2019-11-07

Abstract:Although rapid advances in machine learning have made it increasingly applicable to expert decision-making, the delivery of accurate algorithmic predictions alone is insufficient for effective human-AI collaboration. In this work, we investigate the key types of information medical experts desire when they are first introduced to a diagnostic AI assistant. In a qualitative lab study, we interviewed 21 pathologists before, during, and after being presented deep neural network (DNN) predictions for prostate cancer diagnosis, to learn the types of information that they desired about the AI assistant. Our findings reveal that, far beyond understanding the local, case-specific reasoning behind any model decision, clinicians desired upfront information about basic, global properties of the model, such as its known strengths and limitations, its subjective point-of-view, and its overall design objective--what it's designed to be optimized for. Participants compared these information needs to the collaborative mental models they develop of their medical colleagues when seeking a second opinion: the medical perspectives and standards that those colleagues embody, and the compatibility of those perspectives with their own diagnostic patterns. These findings broaden and enrich discussions surrounding AI transparency for collaborative decision-making, providing a richer understanding of what experts find important in their introduction to AI assistants before integrating them into routine practice.
AI in medical education: a bright new frontier

Andrew Graham

DOI: https://doi.org/10.1136/bmj.q1334

2024-06-22

BMJ

Abstract:In an interview with Bloomberg, Jeremy Stoppelman, chief executive officer of Yelp, observed, "There's been resistance to every new technology that's ever been introduced. When books came out hundreds of years ago, there were complaints that it would destroy the oral tradition. Some of those fears were justified, but it didn't stop the rise of the written word. And books have proven to be incredibly useful."1 It was therefore refreshing to read Morgan and Walsh detailing how The BMJ has sidestepped such resistance and been early adopters of artificial intelligence (AI) as a training adjunct to improve doctors' consultation skills through simulated patient interactions.2There is increasing focus on doctors' non-technical skills and ability to work in a multidisciplinary team. These attributes have been highlighted in the recent update of Good Medical Practice3 and are held in high regard by some national training programmes, which actively score for them in their...

medicine, general & internal
Artificial intelligence (AI) for neurologists: do digital neurones dream of electric sheep?

Joshua Au Yeung,Yang Yang Wang,Zeljko Kraljevic,James T H Teo

DOI: https://doi.org/10.1136/pn-2023-003757

2023-11-17

Practical Neurology

Abstract:Artificial intelligence (AI) is routinely mentioned in journals and newspapers, and non-technical outsiders may have difficulty in distinguishing hyperbole from reality. We present a practical guide to help non-technical neurologists to understand healthcare AI. AI is being used to support clinical decisions in treating neurological disorders. We introduce basic concepts of AI, such as machine learning and natural language processing, and explain how AI is being used in healthcare, giving examples its benefits and challenges. We also cover how AI performance is measured, and its regulatory aspects in healthcare. An important theme is that AI is a general-purpose technology like medical statistics, with broad utility applicable in various scenarios, such that niche approaches are outpaced by approaches that are broadly applicable in many disease areas and specialties. By understanding AI basics and its potential applications, neurologists can make informed decisions when evaluating AI used in their clinical practice. This article was written by four humans, with generative AI helping with formatting and image generation.
Embracing the future—is artificial intelligence already better? A comparative study of artificial intelligence performance in diagnostic accuracy and decision‐making

Ângelo Fonseca,Axel Ferreira,Luís Ribeiro,Sandra Moreira,Cristina Duque

DOI: https://doi.org/10.1111/ene.16195

2024-01-20

European Journal of Neurology

Abstract:Background and purpose The integration of artificial intelligence (AI) in healthcare has the potential to revolutionize patient care and clinical decision‐making. This study aimed to explore the reliability of large language models in neurology by comparing the performance of an AI chatbot with neurologists in diagnostic accuracy and decision‐making. Methods A cross‐sectional observational study was conducted. A pool of clinical cases from the American Academy of Neurology's Question of the Day application was used as the basis for the study. The AI chatbot used was ChatGPT, based on GPT‐3.5. The results were then compared to neurology peers who also answered the questions—a mean of 1500 neurologists/neurology residents. Results The study included 188 questions across 22 different categories. The AI chatbot demonstrated a mean success rate of 71.3% in providing correct answers, with varying levels of proficiency across different neurology categories. Compared to neurology peers, the AI chatbot performed at a similar level, with a mean success rate of 69.2% amongst peers. Additionally, the AI chatbot achieved a correct diagnosis in 85.0% of cases and it provided an adequate justification for its correct responses in 96.1%. Conclusions The study highlights the potential of AI, particularly large language models, in assisting with clinical reasoning and decision‐making in neurology and emphasizes the importance of AI as a complementary tool to human expertise. Future advancements and refinements are needed to enhance the AI chatbot's performance and broaden its application across various medical specialties.

neurosciences,clinical neurology
Helping policy makers and AI companies learn what doctors do

Hongnan Ye

DOI: https://doi.org/10.1136/bmj.q1852

2024-09-04

BMJ

Abstract:Bradley discusses how artificial intelligence (AI) could help ease the technological burden on doctors dealing with current NHS IT systems—a viewpoint that many other medical professionals might have been reluctant to express, perhaps owing to a lack of confidence or opportunity.1 The development of artificial AI applications that are more beneficial for doctors and remove onerous administrative duties is logical. It is therefore surprising that policy makers and providers of AI solutions have not yet taken action in this area. A possible explanation is that neither policy makers nor providers of AI solutions have any comprehension of the typical daily activities of a doctor. In their view, the role of a doctor is to provide treatment to patients, and as a result they are unaware of the additional challenges that doctors face in their work. This is, of course, understandable, given that most people, whether they are staff in the...

medicine, general & internal
Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosis

Federico Cabitza,Andrea Campagner,Luca Ronzio,Matteo Cameli,Giulia Elena Mandoli,Maria Concetta Pastore,Luca Sconfienza,Duarte Folgado,Marília Barandas,Hugo Gamboa

DOI: https://doi.org/10.1016/j.artmed.2023.102506

IF: 7.011

2023-02-09

Artificial Intelligence in Medicine

Abstract:In this paper, we study human-AI collaboration protocols, a design-oriented construct aimed at establishing and evaluating how humans and AI can collaborate in cognitive tasks. We applied this construct in two user studies involving 12 specialist radiologists (the knee MRI study) and 44 ECG readers of varying expertise (the ECG study), who evaluated 240 and 20 cases, respectively, in different collaboration configurations. We confirm the utility of AI support but find that XAI can be associated with a "white-box paradox", producing a null or detrimental effect. We also find that the order of presentation matters: AI-first protocols are associated with higher diagnostic accuracy than human-first protocols, and with higher accuracy than both humans and AI alone. Our findings identify the best conditions for AI to augment human diagnostic skills, rather than trigger dysfunctional responses and cognitive biases that can undermine decision effectiveness.

engineering, biomedical,computer science, artificial intelligence,medical informatics
Patient attitudes toward the AI doctor

Aaron Fanous,Kirsten Steffner,Roxana Daneshjou

DOI: https://doi.org/10.1038/s41591-024-03272-4

IF: 82.9

2024-09-25

Nature Medicine

Abstract:As medical AI development gathers momentum, a new study reveals that much work still needs to be done before the public will willingly embrace AI-based technologies in healthcare.

biochemistry & molecular biology,cell biology,medicine, research & experimental
Performance of Artificial Intelligence (AI)-Powered Chatbots in the Assessment of Medical Case Reports: Qualitative Insights From Simulated Scenarios

Florian Reis,Christian Lenz

DOI: https://doi.org/10.7759/cureus.53899

2024-02-09

Cureus

Abstract:Introduction With the expanding awareness and use of AI-powered chatbots, it seems possible that an increasing number of people could use them to assess and evaluate their medical symptoms. If chatbots are used for this purpose, that have not previously undergone a thorough medical evaluation for this specific use, various risks might arise. The aim of this study is to analyze and compare the performance of popular chatbots in differentiating between severe and less critical medical symptoms described from a patient's perspective and to examine the variations in substantive medical assessment accuracy and empathetic communication style among the chatbots' responses. Materials and methods Our study compared three different AI-supported chatbots - OpenAI's ChatGPT 3.5, Microsoft's Bing Chat, and Inflection's Pi AI. Three exemplary case reports for medical emergencies as well as three cases without an urgent reason for an emergency medical admission were constructed and analyzed. Each case report was accompanied by identical questions concerning the most likely suspected diagnosis and the urgency of an immediate medical evaluation. The respective answers of the chatbots were qualitatively compared with each other regarding the medical accuracy of the differential diagnoses mentioned and the conclusions drawn, as well as regarding patient-oriented and empathetic language. Results All examined chatbots were capable of providing medically plausible and probable diagnoses and classifying situations as acute or less critical. However, their responses varied slightly in the level of their urgency assessment. Clear differences could be seen in the level of detail of the differential diagnoses, the overall length of the answers, and how the chatbot dealt with the challenge of being confronted with medical issues. All given answers were comparable in terms of empathy level and comprehensibility. Conclusion Even AI chatbots that are not designed for medical applications already offer substantial guidance in assessing typical medical emergency indications but should always be provided with a disclaimer. In responding to medical queries, characteristic differences emerge among chatbots in the extent and style of their respective answers. Given the lack of medical supervision of many established chatbots, subsequent studies, and experiences are essential to clarify whether a more extensive use of these chatbots for medical concerns will have a positive impact on healthcare or rather pose major medical risks.
Comparison of artificial intelligence large language model chatbots in answering frequently asked questions in anaesthesia

Teresa P Nguyen,Brendan Carvalho,Hannah Sukhdeo,Kareem Joudi,Nan Guo,Marianne Chen,Jed T Wolpaw,Jesse J Kiefer,Melissa Byrne,Tatiana Jamroz,Allison A Mootz,Sharon C Reale,James Zou,Pervez Sultan

DOI: https://doi.org/10.1016/j.bjao.2024.100280

2024-05-08

BJA Open

Abstract:Background: Patients are increasingly using artificial intelligence (AI) chatbots to seek answers to medical queries. Methods: Ten frequently asked questions in anaesthesia were posed to three AI chatbots: ChatGPT4 (OpenAI), Bard (Google), and Bing Chat (Microsoft). Each chatbot's answers were evaluated in a randomised, blinded order by five residency programme directors from 15 medical institutions in the USA. Three medical content quality categories (accuracy, comprehensiveness, safety) and three communication quality categories (understandability, empathy/respect, and ethics) were scored between 1 and 5 (1 representing worst, 5 representing best). Results: ChatGPT4 and Bard outperformed Bing Chat (median [inter-quartile range] scores: 4 [3-4], 4 [3-4], and 3 [2-4], respectively; P<0.001 with all metrics combined). All AI chatbots performed poorly in accuracy (score of ≥4 by 58%, 48%, and 36% of experts for ChatGPT4, Bard, and Bing Chat, respectively), comprehensiveness (score ≥4 by 42%, 30%, and 12% of experts for ChatGPT4, Bard, and Bing Chat, respectively), and safety (score ≥4 by 50%, 40%, and 28% of experts for ChatGPT4, Bard, and Bing Chat, respectively). Notably, answers from ChatGPT4, Bard, and Bing Chat differed statistically in comprehensiveness (ChatGPT4, 3 [2-4] vs Bing Chat, 2 [2-3], P<0.001; and Bard 3 [2-4] vs Bing Chat, 2 [2-3], P=0.002). All large language model chatbots performed well with no statistical difference for understandability (P=0.24), empathy (P=0.032), and ethics (P=0.465). Conclusions: In answering anaesthesia patient frequently asked questions, the chatbots perform well on communication metrics but are suboptimal for medical content metrics. Overall, ChatGPT4 and Bard were comparable to each other, both outperforming Bing Chat.

Daily briefing: Testers say Google AI has a better bedside manner than human doctors

Google AI has better bedside manner than human doctors — and makes better diagnoses

[Rare disease in the age of artificial intelligence]

Doctor AI? A pilot study examining responses of artificial intelligence to common questions asked by geriatric patients

Following Medical Advice of an AI or a Human Doctor? Experimental Evidence Based on Clinician-Patient Communication Pathway Model

Could artificial intelligence make human doctors obsolete? A survey based on public attitudes

Artificial intelligence powers digital medicine

A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis

Behavioral and neural evidence on consumer responses to human doctors and medical artificial intelligence

Do patients prefer a human doctor, artificial intelligence, or a blend, and is this preference dependent on medical discipline? Empirical evidence and implications for medical practice

When an AI Doctor Gets Personal: The Effects of Social and Medical Individuation in Encounters With Human and AI Doctors

When Doctors Meet with AlphaGo: Potential Application of Machine Learning to Clinical Medicine.

"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making

AI in medical education: a bright new frontier

Artificial intelligence (AI) for neurologists: do digital neurones dream of electric sheep?

Embracing the future—is artificial intelligence already better? A comparative study of artificial intelligence performance in diagnostic accuracy and decision‐making

Helping policy makers and AI companies learn what doctors do

Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosis

Patient attitudes toward the AI doctor

Performance of Artificial Intelligence (AI)-Powered Chatbots in the Assessment of Medical Case Reports: Qualitative Insights From Simulated Scenarios

Comparison of artificial intelligence large language model chatbots in answering frequently asked questions in anaesthesia