Evaluating Artificial Intelligence Responses to Public Health Questions

Zechariah Zhu,Adam Poliak,Mark Dredze,Michael Hogarth,John W. Ayers,Eric C. Leas,Davey M. Smith

DOI: https://doi.org/10.1001/jamanetworkopen.2023.17517

2023-06-09

JAMA Network Open

Abstract:This cross-sectional study analyzes the quality of ChatGPT responses to public health questions.

medicine, general & internal

What problem does this paper attempt to address?

Artificial Intelligence and Public Health: Evaluating ChatGPT Responses to Vaccination Myths and Misconceptions

Giovanna Deiana,Marco Dettori,Antonella Arghittu,Antonio Azara,Giovanni Gabutti,Paolo Castiglia

DOI: https://doi.org/10.3390/vaccines11071217

2023-07-08

Vaccines

Abstract:Artificial intelligence (AI), such as ChatGPT, is the subject of intense debate regarding its possible applications, including in the health care context. This study evaluates Correctness, Clarity, and Exhaustiveness of the answers provided by ChatGPT on the topic of vaccination. The World Health Organization's 11 "myths and misconceptions" about vaccinations were administered to both the free (GPT-3.5) and paid version (GPT-4.0) of ChatGPT. The AI's responses were evaluated qualitatively and quantitatively, in reference to those provided by WHO, independently by two expert Raters. The agreement between the Raters was significant for both versions (p of K < 0.05). Overall, ChatGPT responses were easy to understand and 85.4% accurate although one of the questions was misinterpreted. Qualitatively, GPT-4.0 responses were superior to GPT-3.5 responses in Correctness, Clarity, and Exhaustiveness (Δ = 5.6%, 17.9%, 9.3% respectively). The study shows that, if appropriately questioned, AIs can represent a useful aid in the health care field. However, when consulted by non-expert users, without the support of expert medical advice, it is not free from the risk of eliciting misleading responses. Moreover, given the existing social divide in information access, the improved accuracy of answers from the paid version raises further ethical issues.

immunology,medicine, research & experimental
Artificial Intelligence and Public Health: An Exploratory Study

D. Haluza,David Jungwirth

DOI: https://doi.org/10.3390/ijerph20054541

IF: 4.614

2023-03-01

International Journal of Environmental Research and Public Health

Abstract:Artificial intelligence (AI) has the potential to revolutionize research by automating data analysis, generating new insights, and supporting the discovery of new knowledge. The top 10 contribution areas of AI towards public health were gathered in this exploratory study. We utilized the “text-davinci-003” model of GPT-3, using OpenAI playground default parameters. The model was trained with the largest training dataset any AI had, limited to a cut-off date in 2021. This study aimed to test the ability of GPT-3 to advance public health and to explore the feasibility of using AI as a scientific co-author. We asked the AI asked for structured input, including scientific quotations, and reviewed responses for plausibility. We found that GPT-3 was able to assemble, summarize, and generate plausible text blocks relevant for public health concerns, elucidating valuable areas of application for itself. However, most quotations were purely invented by GPT-3 and thus invalid. Our research showed that AI can contribute to public health research as a team member. According to authorship guidelines, the AI was ultimately not listed as a co-author, as it would be done with a human researcher. We conclude that good scientific practice also needs to be followed for AI contributions, and a broad scientific discourse on AI contributions is needed.

Medicine,Computer Science
Can ChatGPT follow an algorithm for ethical decision-making in public health?

A Anov,K Statev,A Stateva,A Seizov,S Aleksandrova-Yankulovska

DOI: https://doi.org/10.1093/eurpub/ckad160.1284

2023-10-01

European Journal of Public Health

Abstract:Abstract Background With the rapid development of AI, especially chat bots based on Large Language Models like OpenAI ChatGPT and Google Bard, the question of the day is can AI replace (partially or entirely) experts in certain fields - for example in Medical Ethics. The aim of this work is to test the application of ChatGPT to public health ethics (PHE) cases. Methods ChatGPT was given the task to analyse Ebola quarantine case in Sierra Leone by applying methodology for ethical analysis of its own choice and by applying a 4-step model for ethical analysis developed in Medical University - Pleven. The tests were done with simple and detailed tasks. No additional plugins to ChatGPT were included. Results Given a simple task ChatGPT applied 4-step model to the case - Facts, Stakeholders, Ethical principles (Autonomy, Non-maleficence, Beneficence, Justice), Potential consequences. A brief argumentation was provided for each point. ChatGPT identified key ethical concepts for the case (limiting autonomy and freedom, limiting access to resources). When asked to analyse the case with the principles of PHE as defined in The Cambridge Textbook of Bioethics, ChatGPT still used Beauchamp and Childress principles of bioethics. When introduced to PHE principles ChatGPT identified limiting access to resources, creating social and economic hardship for those affected by quarantine as key issues. ChatGPT managed to apply the 4-step model for ethical analysis developed in Medical University - Pleven to some degree. It produced detailed discussion but didn't use any ethical standards, patient's rights or public health policies that could help with decision-making. Individual autonomy was disregarded by ChatGPT in the most detailed analysis. Conclusions ChatGPT has basic knowledge of methodologies for ethical case analysis. With proper guidance and detailed instructions, it could follow complex methodologies and produce a decision. Key messages • AI will continue to develop and its inclusion in medicine and public health practice and teaching is inevitable. • Public health professionals should always be critical of AI decisions and double check the information that was provided by AI.

public, environmental & occupational health
Artificial intelligence in global health equity: an evaluation and discussion on the application of ChatGPT, in the Chinese National Medical Licensing Examination

Wenting Tong,Yongfu Guan,Jinping Chen,Xixuan Huang,Yuting Zhong,Changrong Zhang,Hui Zhang

DOI: https://doi.org/10.3389/fmed.2023.1237432

2023-10-19

Abstract:Background: The demand for healthcare is increasing globally, with notable disparities in access to resources, especially in Asia, Africa, and Latin America. The rapid development of Artificial Intelligence (AI) technologies, such as OpenAI's ChatGPT, has shown promise in revolutionizing healthcare. However, potential challenges, including the need for specialized medical training, privacy concerns, and language bias, require attention. Methods: To assess the applicability and limitations of ChatGPT in Chinese and English settings, we designed an experiment evaluating its performance in the 2022 National Medical Licensing Examination (NMLE) in China. For a standardized evaluation, we used the comprehensive written part of the NMLE, translated into English by a bilingual expert. All questions were input into ChatGPT, which provided answers and reasons for choosing them. Responses were evaluated for "information quality" using the Likert scale. Results: ChatGPT demonstrated a correct response rate of 81.25% for Chinese and 86.25% for English questions. Logistic regression analysis showed that neither the difficulty nor the subject matter of the questions was a significant factor in AI errors. The Brier Scores, indicating predictive accuracy, were 0.19 for Chinese and 0.14 for English, indicating good predictive performance. The average quality score for English responses was excellent (4.43 point), slightly higher than for Chinese (4.34 point). Conclusion: While AI language models like ChatGPT show promise for global healthcare, language bias is a key challenge. Ensuring that such technologies are robustly trained and sensitive to multiple languages and cultures is vital. Further research into AI's role in healthcare, particularly in areas with limited resources, is warranted.
ChatGPT vs Medical Professional: Analyzing Responses to Laboratory Medicine Questions on Social Media

Mark R Girton,Dina N Greene,Geralyn Messerlian,David F Keren,Min Yu

DOI: https://doi.org/10.1093/clinchem/hvae093

IF: 12.114

2024-07-16

Clinical Chemistry

Abstract:Abstract Background The integration of ChatGPT, a large language model (LLM) developed by OpenAI, into healthcare has sparked significant interest due to its potential to enhance patient care and medical education. With the increasing trend of patients accessing laboratory results online, there is a pressing need to evaluate the effectiveness of ChatGPT in providing accurate laboratory medicine information. Our study evaluates ChatGPT's effectiveness in addressing patient questions in this area, comparing its performance with that of medical professionals on social media. Methods This study sourced patient questions and medical professional responses from Reddit and Quora, comparing them with responses generated by ChatGPT versions 3.5 and 4.0. Experienced laboratory medicine professionals evaluated the responses for quality and preference. Evaluation results were further analyzed using R software. Results The study analyzed 49 questions, with evaluators reviewing responses from both medical professionals and ChatGPT. ChatGPT's responses were preferred by 75.9% of evaluators and generally received higher ratings for quality. They were noted for their comprehensive and accurate information, whereas responses from medical professionals were valued for their conciseness. The interrater agreement was fair, indicating some subjectivity but a consistent preference for ChatGPT's detailed responses. Conclusions ChatGPT demonstrates potential as an effective tool for addressing queries in laboratory medicine, often surpassing medical professionals in response quality. These results support the need for further research to confirm ChatGPT's utility and explore its integration into healthcare settings.

medical laboratory technology
Assessing ChatGPT's Responses to Otolaryngology Patient Questions

Jonathan M. Carnino,William R. Pellegrini,Megan Willis,Michael B. Cohen,Marianella Paz-Lansberg,Elizabeth M. Davis,Gregory A. Grillone,Jessica R. Levi

DOI: https://doi.org/10.1177/00034894241249621

2024-04-28

Annals of Otology Rhinology & Laryngology

Abstract:Annals of Otology, Rhinology &Laryngology, Ahead of Print. Objective:This study aims to evaluate ChatGPT's performance in addressing real-world otolaryngology patient questions, focusing on accuracy, comprehensiveness, and patient safety, to assess its suitability for integration into healthcare.Methods:A cross-sectional study was conducted using patient questions from the public online forum Reddit's r/AskDocs, where medical advice is sought from healthcare professionals. Patient questions were input into ChatGPT (GPT-3.5), and responses were reviewed by 5 board-certified otolaryngologists. The evaluation criteria included difficulty, accuracy, comprehensiveness, and bedside manner/empathy. Statistical analysis explored the relationship between patient question characteristics and ChatGPT response scores. Potentially dangerous responses were also identified.Results:Patient questions averaged 224.93 words, while ChatGPT responses were longer at 414.93 words. The accuracy scores for ChatGPT responses were 3.76/5, comprehensiveness scores were 3.59/5, and bedside manner/empathy scores were 4.28/5. Longer patient questions did not correlate with higher response ratings. However, longer ChatGPT responses scored higher in bedside manner/empathy. Higher question difficulty correlated with lower comprehensiveness. Five responses were flagged as potentially dangerous.Conclusion:While ChatGPT exhibits promise in addressing otolaryngology patient questions, this study demonstrates its limitations, particularly in accuracy and comprehensiveness. The identification of potentially dangerous responses underscores the need for a cautious approach to AI in medical advice. Responsible integration of AI into healthcare necessitates thorough assessments of model performance and ethical considerations for patient safety.

otorhinolaryngology
Can ChatGPT help patients answer their otolaryngology questions?

Habib G. Zalzal,Ariel Abraham,Jenhao Cheng,Rahul K. Shah

DOI: https://doi.org/10.1002/lio2.1193

IF: 2.542

2023-12-09

Laryngoscope Investigative Otolaryngology

Abstract:Abstract Background Over the past year, the world has been captivated by the potential of artificial intelligence (AI). The appetite for AI in science, specifically healthcare is huge. It is imperative to understand the credibility of large language models in assisting the public in medical queries. Objective To evaluate the ability of ChatGPT to provide reasonably accurate answers to public queries within the domain of Otolaryngology. Methods Two board‐certified otolaryngologists (HZ, RS) inputted 30 text‐based patient queries into the ChatGPT‐3.5 model. ChatGPT responses were rated by physicians on a scale (accurate, partially accurate, incorrect), while a similar 3‐point scale involving confidence was given to layperson reviewers. Demographic data involving gender and education level was recorded for the public reviewers. Inter‐rater agreement percentage was based on binomial distribution for calculating the 95% confidence intervals and performing significance tests. Statistical significance was defined as p < .05 for two‐sided tests. Results In testing patient queries, both Otolaryngology physicians found that ChatGPT answered 98.3% of questions correctly, but only 79.8% (range 51.7%–100%) of patients were confident that the AI model was accurate in its responses (corrected agreement = 0.682; p < .001). Among the layperson responses, the corrected coefficient was of moderate agreement (0.571; p < .001). No correlation was noted among age, gender, or education level for the layperson responses. Conclusion ChatGPT is highly accurate in responding to questions posed by the public with regards to Otolaryngology from a physician standpoint. Public reviewers were not fully confident in believing the AI model, with subjective concerns related to less trust in AI answers compared to physician explanation. Larger evaluations with a representative public sample and broader medical questions should immediately be conducted by appropriate organizations, governing bodies, and/or governmental agencies to instill public confidence in AI and ChatGPT as a medical resource. Level of Evidence 4.

otorhinolaryngology
Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example

Lu, Yanqiu,Gu, Shuqin

DOI: https://doi.org/10.1007/s10439-023-03203-3

IF: 3.8

2023-04-20

Annals of Biomedical Engineering

Abstract:Over the past month, a new AI model called Chatbot Generative Pre-trained Transformer (ChatGPT), has received enormous attention in the media and scientific communities due to its ability to process and respond to commands in a humanistic fashion. As reported, five days after its launch, the number of registered users of ChatGPT exceeded one million, and its monthly active users had exceeded 100 million two months later, making it the most rapidly growing consumer application in history. The advent of ChatGPT has further brought about new ideas and challenges in the realm of infectious disease. In view of this, in order to evaluate the potential use of ChatGPT in clinical practice and scientific research of infectious disease, we conducted a brief online survey by using the publicly available ChatGPT webpage. Also, the present study also talks about the relevant social and ethical issues related to this program.

engineering, biomedical
Accuracy of Online Artificial Intelligence Models in Primary Care Settings

Joseph Kassab,Abdel Hadi El Hajjar,Richard M Wardrop,Andrei Brateanu,Abdel Hadi el Hajjar,Richard M. Wardrop

DOI: https://doi.org/10.1016/j.amepre.2024.02.006

IF: 6.604

2024-02-01

American Journal of Preventive Medicine

Abstract:INTRODUCTION: The importance of preventive medicine and primary care in the sphere of public health is expanding, yet a gap exists in the utilization of recommended medical services. As patients increasingly turn to online resources for supplementary advice, the role of artificial intelligence (AI) in providing accurate and reliable information has emerged. The present study aimed to assess ChatGPT-4's and Google Bard's capacity to deliver accurate recommendations in preventive medicine and primary care.METHODS: Fifty-six questions were formulated and presented to ChatGPT-4 in June 2023 and Google Bard in October 2023, and the responses were independently reviewed by two physicians, with each answer being classified as "accurate," "inaccurate," or "accurate with missing information." Disagreements were resolved by a third physician.RESULTS: Initial inter-reviewer agreement on grading was substantial (Cohen's Kappa was 0.76, 95%CI [0.61-0.90] for ChatGPT-4 and 0.89, 95%CI [0.79-0.99] for Bard). After reaching a consensus, 28.6% of ChatGPT-4-generated answers were deemed accurate, 28.6% inaccurate, and 42.8% accurate with missing information. In comparison, 53.6% of Bard-generated answers were deemed accurate, 17.8% inaccurate, and 28.6% accurate with missing information. Responses to CDC and immunization-related questions showed notable inaccuracies (80%) in both models.CONCLUSIONS: ChatGPT-4 and Bard demonstrated potential in offering accurate information in preventive care. It also brought to light the critical need for regular updates, particularly in the rapidly evolving areas of medicine. A significant proportion of the AI models' responses were deemed "accurate with missing information," emphasizing the importance of viewing AI tools as complementary resources when seeking medical information.

medicine, general & internal,public, environmental & occupational health
Comparing the quality of ChatGPT- and physician-generated responses to patients’ dermatology questions in the electronic medical record

Kelly Reynolds,Daniel Nadelman,Joseph Durgin,Stephen Ansah-Addo,Daniel Cole,Rachel Fayne,Jane Harrell,Madison Ratycz,Mason Runge,Amanda Shepard-Hayes,Daniel Wenzel,Trilokraj Tejasvi

DOI: https://doi.org/10.1093/ced/llad456

2024-01-05

Clinical and Experimental Dermatology

Abstract:Abstract Background ChatGPT is a free artificial intelligence (AI)-based natural language processing tool that generates complex responses to inputs from users. Objectives To determine whether ChatGPT is able to generate high-quality responses to patient-submitted questions in the patient portal. Methods Patient-submitted questions and the corresponding responses from their dermatology physician were extracted from the electronic medical record for analysis. The questions were input into ChatGPT (version 3.5) and the outputs extracted for analysis, with manual removal of verbiage pertaining to ChatGPT’s inability to provide medical advice. Ten blinded reviewers (seven physicians and three nonphysicians) rated and selected their preference in terms of ‘overall quality’, ‘readability’, ‘accuracy’, ‘thoroughness’ and ‘level of empathy’ of the physician- and ChatGPT-generated responses. Results Thirty-one messages and responses were analysed. Physician-generated responses were vastly preferred over the ChatGPT responses by the physician and nonphysician reviewers and received significantly higher ratings for ‘readability’ and ‘level of empathy’. Conclusions The results of this study suggest that physician-generated responses to patients’ portal messages are still preferred over ChatGPT, but generative AI tools may be helpful in generating the first drafts of responses and providing information on education resources for patients.

dermatology
Assessing the Quality and Reliability of AI-Generated Responses to Common Hypertension Queries

Aleena Vinufrancis,Hussein Al Hussein,Heena V Patel,Afshan Nizami,Aditya Singh,Bianca Nunez,Aiah Mounir Abdel-Aal

DOI: https://doi.org/10.7759/cureus.66041

2024-08-02

Cureus

Abstract:Introduction: The integration of artificial intelligence (AI) in healthcare, particularly through language models like ChatGPT and ChatSonic, has gained substantial attention. This article explores the utilization of these AI models to address patient queries related to hypertension, emphasizing their potential to enhance health literacy and disease understanding. The study aims to compare the quality and reliability of responses generated by ChatGPT and ChatSonic in addressing common patient queries about hypertension and evaluate these AI models using the Global Quality Scale (GQS) and the Modified DISCERN scale. Methods: A virtual cross-sectional observational study was conducted over one month, starting in October 2023. Ten common patient queries regarding hypertension were presented to ChatGPT (https://chat.openai.com/) and ChatSonic (https://writesonic.com/chat), and the responses were recorded. Two internal medicine physicians assessed the responses using the GQS and the Modified DISCERN scale. Statistical analysis included Cohen's Kappa values for inter-rater agreement. Results: The study evaluated responses from ChatGPT and ChatSonic for 10 patient queries. Assessors observed variations in the quality and reliability assessments between the two AI models. Cohen's Kappa values indicated minimal agreement between the evaluators for both the GQS and Modified DISCERN scale. Conclusions: This study highlights the variations in the assessment of responses generated by ChatGPT and ChatSonic for hypertension-related queries. The findings highlight the need for ongoing monitoring and fact-checking of AI-generated responses.
A Comparative Analysis of Large Language Models on Clinical Questions for Autoimmune Diseases

Weiming Zhang,Jie Yu,Juntao Ma,Jiawei Feng,Linyu Geng,Yuxin Chen,Huayong Zhang,Mingzhe Ning

DOI: https://doi.org/10.21203/rs.3.rs-4810651/v1

2024-01-01

Abstract:Background Artificial intelligence (AI) has made great strides. Our study evaluated the performance in delivering clinical questions related to autoimmune diseases (AIDs). Methods 46 AIDs-related questions were compiled and entered into ChatGPT 3.5, ChatGPT 4.0, and Gemini. The replies were collected and sent to laboratory specialists for scoring according to relevance, correctness, completeness, helpfulness, and safety. Scores for three chatbots in five quality dimensions and the scores of the replies to the questions under each quality dimension were analyzed. Results ChatGPT 4.0 showed superior performance than ChatGPT 3.5 and Gemini in all five quality dimensions. ChatGPT 4.0 outperformed ChatGPT 3.5 or Gemini on the relevance, completeness or helpfulness in answering about the prognosis, diagnosis, or the report interpretation of AIDs. ChatGPT 4.0’s replies were the longest, followed by ChatGPT 3.5, Gemini’s was the shortest. Conclusions Our findings highlight ChatGPT 4.0 is superior to delivering comprehensive and accurate responses to AIDs-related clinical questions.
"Doctor ChatGPT, Can You Help Me?" The Patient's Perspective: Cross-Sectional Study

Jonas Armbruster,Florian Bussmann,Catharina Rothhaas,Nadine Titze,Paul Alfred Grützner,Holger Freischmidt

DOI: https://doi.org/10.2196/58831

IF: 7.076

2024-10-02

Journal of Medical Internet Research

Abstract:Background: Artificial intelligence and the language models derived from it, such as ChatGPT, offer immense possibilities, particularly in the field of medicine. It is already evident that ChatGPT can provide adequate and, in some cases, expert-level responses to health-related queries and advice for patients. However, it is currently unknown how patients perceive these capabilities, whether they can derive benefit from them, and whether potential risks, such as harmful suggestions, are detected by patients. Objective: This study aims to clarify whether patients can get useful and safe health care advice from an artificial intelligence chatbot assistant. Methods: This cross-sectional study was conducted using 100 publicly available health-related questions from 5 medical specialties (trauma, general surgery, otolaryngology, pediatrics, and internal medicine) from a web-based platform for patients. Responses generated by ChatGPT-4.0 and by an expert panel (EP) of experienced physicians from the aforementioned web-based platform were packed into 10 sets consisting of 10 questions each. The blinded evaluation was carried out by patients regarding empathy and usefulness (assessed through the question: "Would this answer have helped you?") on a scale from 1 to 5. As a control, evaluation was also performed by 3 physicians in each respective medical specialty, who were additionally asked about the potential harm of the response and its correctness. Results: In total, 200 sets of questions were submitted by 64 patients (mean 45.7, SD 15.9 years; 29/64, 45.3% male), resulting in 2000 evaluated answers of ChatGPT and the EP each. ChatGPT scored higher in terms of empathy (4.18 vs 2.7; P PP=.049). Ratings of ChatGPT were high regardless of the participant's age. The same highly significant results were observed in the evaluation of the respective specialist physicians. ChatGPT outperformed significantly in correctness (4.51 vs 3.55; P P

health care sciences & services,medical informatics
Generative artificial intelligence models in clinical infectious disease consultations: a cross-sectional analysis among specialists and resident trainees

Edwin Kwan-Yeung Chiu,Siddharth Sridhar,Samson Sai-Yin Wong,Anthony Raymond Tam,Ming-Hong Choi,Alicia Wing-Tung Lau,Wai-Ching Wong,Kelvin Hei-Yeung Chiu,Yuey-Zhun Ng,Kwok-Yung Yuen,Tom Wai-Hin Chung

DOI: https://doi.org/10.1101/2024.08.15.24312054

2024-08-19

Abstract:Background The potential of generative artificial intelligence (GenAI) to augment clinical consultation services in clinical microbiology and infectious diseases (ID) is being evaluated. Methods This cross-sectional study evaluated the performance of four GenAI chatbots (GPT-4.0, a Custom Chatbot based on GPT-4.0, Gemini Pro, and Claude 2) by analysing 40 unique clinical scenarios synthesised from real-life clinical notes. Six specialists and resident trainees from clinical microbiology or ID units conducted randomised, blinded evaluations across four key domains: factual consistency, comprehensiveness, coherence, and medical harmfulness. Results Analysis of 960 human evaluation entries by six clinicians, covering 160 AI-generated responses, showed that GPT-4.0 produced longer responses than Gemini Pro (p<0·001) and Claude 2 (p<0·001), averaging 577 ±81·19 words. GPT-4.0 achieved significantly higher mean composite scores compared to Gemini Pro [mean difference (MD)=0·2313, p=0·001] and Claude 2 (MD=0·2021, p=0·006). Specifically, GPT-4.0 outperformed Gemini Pro and Claude 2 in factual consistency (Gemini Pro, p=0·02 Claude 2, p=0·02), comprehensiveness (Gemini Pro, p=0·04; Claude 2, p=0·03), and the absence of medical harm (Gemini Pro, p=0·02; Claude 2, p=0·04). Within-group comparisons showed that specialists consistently awarded higher ratings than resident trainees across all assessed domains (p<0·001) and overall composite scores (p<0·001). Specialists were 9 times more likely to recognise responses with "Fully verified facts" and 5 times more likely to consider responses as "Harmless". However, post-hoc analysis revealed that specialists may inadvertently disregard conflicting or inaccurate information in their assessments, thereby erroneously assigning higher scores. Interpretation Clinical experience and domain expertise of individual clinicians significantly shaped the interpretation of AI-generated responses. In our analysis, we have demonstrated disconcerting human vulnerabilities in safeguarding against potentially harmful outputs. This fallibility seemed to be most apparent among experienced specialists and domain experts, revealing an unsettling paradox in the human evaluation and oversight of advanced AI systems. Stakeholders and developers must strive to control and mitigate user-specific and cognitive biases, thereby maximising the clinical impact and utility of AI technologies in healthcare delivery.

Infectious Diseases (except HIV/AIDS)
A Clinical Evaluation of Cardiovascular Emergencies: A Comparison of Responses from ChatGPT, Emergency Physicians, and Cardiologists

Muhammet Geneş,Bülent Deveci

DOI: https://doi.org/10.3390/diagnostics14232731

IF: 3.6

2024-12-06

Diagnostics

Abstract:Background: Artificial intelligence (AI) tools, like ChatGPT, are gaining attention for their potential in supporting clinical decisions. This study evaluates the performance of ChatGPT-4o in acute cardiological cases compared to cardiologists and emergency physicians. Methods: Twenty acute cardiological scenarios were used to compare the responses of ChatGPT-4o, cardiologists, and emergency physicians in terms of accuracy, completeness, and response time. Statistical analyses included the Kruskal–Wallis H test and post hoc comparisons using the Mann–Whitney U test with Bonferroni correction. Results: ChatGPT-4o and cardiologists both achieved 100% correct response rates, while emergency physicians showed lower accuracy. ChatGPT-4o provided the fastest responses and obtained the highest accuracy and completeness scores. Statistically significant differences were found between ChatGPT-4o and emergency physicians (p < 0.001), and between cardiologists and emergency physicians (p < 0.001). A Cohen's kappa value of 0.92 indicated a high level of inter-rater agreement. Conclusions: ChatGPT-4o outperformed human clinicians in accuracy, completeness, and response time, highlighting its potential as a clinical decision support tool. However, human oversight remains essential to ensure safe AI integration in healthcare settings.

medicine, general & internal
Role of Chat GPT in Public Health

Biswas, Som S.

DOI: https://doi.org/10.1007/s10439-023-03172-7

IF: 3.8

2023-03-16

Annals of Biomedical Engineering

Abstract:ChatGPT, a language model developed by OpenAI, has the potential to play a role in public health. With its ability to generate human-like text based on large amounts of data, ChatGPT has the potential to support individuals and communities in making informed decisions about their health (Panch et al. Lancet Digit Health 1:e13–e14, 2019; Baclic et al. Canada Commun Dis Rep 46.6:161, 2020). However, as with any technology, there are limitations and challenges to consider when using ChatGPT in public health. In this overview, we will examine the potential uses of ChatGPT in public health, as well as the advantages and disadvantages of its use.

engineering, biomedical
Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice

Anton Danholt Lautrup,Tobias Hyrup,Anna Schneider-Kamp,Marie Dahl,Jes Sanddal Lindholt,Peter Schneider-Kamp

DOI: https://doi.org/10.1136/openhrt-2023-002455

Open Heart

Abstract:Objectives: The advent of conversational artificial intelligence (AI) systems employing large language models such as ChatGPT has sparked public, professional and academic debates on the capabilities of such technologies. This mixed-methods study sets out to review and systematically explore the capabilities of ChatGPT to adequately provide health advice to patients when prompted regarding four topics from the field of cardiovascular diseases. Methods: As of 30 May 2023, 528 items on PubMed contained the term ChatGPT in their title and/or abstract, with 258 being classified as journal articles and included in our thematic state-of-the-art review. For the experimental part, we systematically developed and assessed 123 prompts across the four topics based on three classes of users and two languages. Medical and communications experts scored ChatGPT's responses according to the 4Cs of language model evaluation proposed in this article: correct, concise, comprehensive and comprehensible. Results: The articles reviewed were fairly evenly distributed across discussing how ChatGPT could be used for medical publishing, in clinical practice and for education of medical personnel and/or patients. Quantitatively and qualitatively assessing the capability of ChatGPT on the 123 prompts demonstrated that, while the responses generally received above-average scores, they occupy a spectrum from the concise and correct via the absurd to what only can be described as hazardously incorrect and incomplete. Prompts formulated at higher levels of health literacy generally yielded higher-quality answers. Counterintuitively, responses in a lower-resource language were often of higher quality. Conclusions: The results emphasise the relationship between prompt and response quality and hint at potentially concerning futures in personalised medicine. The widespread use of large language models for health advice might amplify existing health inequalities and will increase the pressure on healthcare systems by providing easy access to many seemingly likely differential diagnoses and recommendations for seeing a doctor for even harmless ailments.
Who is asking ChatGPT health questions? Analysis of a nationally representative Australian community sample

Julie Ayre,Erin Cvejic,Kirsten J McCaffery

DOI: https://doi.org/10.1101/2024.10.13.24315426

2024-10-15

Abstract:ChatGPT and other generative AI tools have dramatically changed the health information environment. Discussions about their benefits and risks need clear evidence about who is using them and why. This nationally representative survey estimates that one in ten Australians asked ChatGPT a health question in the past 6 months (95%CI: 8.5%-11.4%), with higher use in key priority groups: people born in a non-English speaking country, who speak a non-English language at home, and who have low health literacy (p's <0.05). As usage increases, we need to equip community with knowledge and skills to use these tools safely.
AI in obstetrics: Evaluating residents' capabilities and interaction strategies with ChatGPT

David Desseauve,Raphael Lescar,Benoit de la Fourniere,Pierre-François Ceccaldi,Mikhail Dziadzko

DOI: https://doi.org/10.1016/j.ejogrb.2024.09.008

Abstract:In line with the digital transformation trend in medical training, students may resort to artificial intelligence (AI) for learning. This study assessed the interaction between obstetrics residents and ChatGPT during clinically oriented summative evaluations related to acute hepatic steatosis of pregnancy, and their self-reported competencies in information technology (IT) and AI. The participants in this semi-qualitative observational study were 14 obstetrics residents from two university hospitals. Students' queries were categorized into three distinct types: third-party enquiries; search-engine-style queries; and GPT-centric prompts. Responses were compared against a standardized answer produced by ChatGPT with a Delphi-developed expert prompt. Data analysis employed descriptive statistics and correlation analysis to explore the relationship between AI/IT skills and response accuracy. The study participants showed moderate IT proficiency but low AI proficiency. Interaction with ChatGPT regarding clinical signs of acute hepatic steatosis gravidarum revealed a preference for third-party questioning, resulting in only 21% accurate responses due to misinterpretation of medical acronyms. No correlation was found between AI response accuracy and the residents' self-assessed IT or AI skills, with most expressing dissatisfaction with their AI training. This study underlines the discrepancy between perceived and actual AI proficiency, highlighted by clinically inaccurate yet plausible AI responses - a manifestation of the 'stochastic parrot' phenomenon. These findings advocate for the inclusion of structured AI literacy programmes in medical education, focusing on prompt engineering. These academic skills are essential to exploit AI's potential in obstetrics and gynaecology. The ultimate aim is to optimize patient care in AI-augmented health care, and prevent misleading and unsafe knowledge acquisition.
Assessing the Accuracy, Completeness, and Reliability of Artificial Intelligence-Generated Responses in Dentistry: A Pilot Study Evaluating the ChatGPT Model

Kelly F Molena,Ana P Macedo,Anum Ijaz,Fabrício K Carvalho,Maria Julia D Gallo,Francisco Wanderley Garcia de Paula E Silva,Andiara de Rossi,Luis A Mezzomo,Leda Regina F Mugayar,Alexandra M Queiroz

DOI: https://doi.org/10.7759/cureus.65658

2024-07-29

Cureus

Abstract:Background: Artificial intelligence (AI) can be a tool in the diagnosis and acquisition of knowledge, particularly in dentistry, sparking debates on its application in clinical decision-making. Objective: This study aims to evaluate the accuracy, completeness, and reliability of the responses generated by Chatbot Generative Pre-Trained Transformer (ChatGPT) 3.5 in dentistry using expert-formulated questions. Materials and methods: Experts were invited to create three questions, answers, and respective references according to specialized fields of activity. The Likert scale was used to evaluate agreement levels between experts and ChatGPT responses. Statistical analysis compared descriptive and binary question groups in terms of accuracy and completeness. Questions with low accuracy underwent re-evaluation, and subsequent responses were compared for improvement. The Wilcoxon test was utilized (α = 0.05). Results: Ten experts across six dental specialties generated 30 binary and descriptive dental questions and references. The accuracy score had a median of 5.50 and a mean of 4.17. For completeness, the median was 2.00 and the mean was 2.07. No difference was observed between descriptive and binary responses for accuracy and completeness. However, re-evaluated responses showed a significant improvement with a significant difference in accuracy (median 5.50 vs. 6.00; mean 4.17 vs. 4.80; p=0.042) and completeness (median 2.0 vs. 2.0; mean 2.07 vs. 2.30; p=0.011). References were more incorrect than correct, with no differences between descriptive and binary questions. Conclusions: ChatGPT initially demonstrated good accuracy and completeness, which was further improved with machine learning (ML) over time. However, some inaccurate answers and references persisted. Human critical discernment continues to be essential to facing complex clinical cases and advancing theoretical knowledge and evidence-based practice.

Evaluating Artificial Intelligence Responses to Public Health Questions

Artificial Intelligence and Public Health: Evaluating ChatGPT Responses to Vaccination Myths and Misconceptions

Artificial Intelligence and Public Health: An Exploratory Study

Can ChatGPT follow an algorithm for ethical decision-making in public health?

Artificial intelligence in global health equity: an evaluation and discussion on the application of ChatGPT, in the Chinese National Medical Licensing Examination

ChatGPT vs Medical Professional: Analyzing Responses to Laboratory Medicine Questions on Social Media

Assessing ChatGPT's Responses to Otolaryngology Patient Questions

Can ChatGPT help patients answer their otolaryngology questions?

Potential Use of Artificial Intelligence in Infectious Disease: Take ChatGPT as an Example

Accuracy of Online Artificial Intelligence Models in Primary Care Settings

Comparing the quality of ChatGPT- and physician-generated responses to patients’ dermatology questions in the electronic medical record

Assessing the Quality and Reliability of AI-Generated Responses to Common Hypertension Queries

A Comparative Analysis of Large Language Models on Clinical Questions for Autoimmune Diseases

"Doctor ChatGPT, Can You Help Me?" The Patient's Perspective: Cross-Sectional Study

Generative artificial intelligence models in clinical infectious disease consultations: a cross-sectional analysis among specialists and resident trainees

A Clinical Evaluation of Cardiovascular Emergencies: A Comparison of Responses from ChatGPT, Emergency Physicians, and Cardiologists

Role of Chat GPT in Public Health

Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice

Who is asking ChatGPT health questions? Analysis of a nationally representative Australian community sample

AI in obstetrics: Evaluating residents' capabilities and interaction strategies with ChatGPT

Assessing the Accuracy, Completeness, and Reliability of Artificial Intelligence-Generated Responses in Dentistry: A Pilot Study Evaluating the ChatGPT Model