Reliability and accuracy of artificial intelligence ChatGPT in providing information on ophthalmic diseases and management to patients

Francesco Cappellani,Kevin R. Card,Carol L. Shields,Jose S. Pulido,Julia A. Haller

DOI: https://doi.org/10.1038/s41433-023-02906-0

IF: 4.4563

2024-01-21

Eye

Abstract:To assess the accuracy of ophthalmic information provided by an artificial intelligence chatbot (ChatGPT).

ophthalmology

What problem does this paper attempt to address?

Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images

Andrew Mihalache,Ryan S. Huang,Marko M. Popovic,Nikhil S. Patil,Bhadra U. Pandya,Reut Shor,Austin Pereira,Jason M. Kwok,Peng Yan,David T. Wong,Peter J. Kertes,Rajeev H. Muni

DOI: https://doi.org/10.1001/jamaophthalmol.2024.0017

IF: 8.253

2024-02-29

JAMA Ophthalmology

Abstract:Importance Ophthalmology is reliant on effective interpretation of multimodal imaging to ensure diagnostic accuracy. The new ability of ChatGPT-4 (OpenAI) to interpret ophthalmic images has not yet been explored. Objective To evaluate the performance of the novel release of an artificial intelligence chatbot that is capable of processing imaging data. Design, Setting, and Participants This cross-sectional study used a publicly available dataset of ophthalmic cases from OCTCases, a medical education platform based out of the Department of Ophthalmology and Vision Sciences at the University of Toronto, with accompanying clinical multimodal imaging and multiple-choice questions. Across 137 available cases, 136 contained multiple-choice questions (99%). Exposures The chatbot answered questions requiring multimodal input from October 16 to October 23, 2023. Main Outcomes and Measures The primary outcome was the accuracy of the chatbot in answering multiple-choice questions pertaining to image recognition in ophthalmic cases, measured as the proportion of correct responses. χ 2 Tests were conducted to compare the proportion of correct responses across different ophthalmic subspecialties. Results A total of 429 multiple-choice questions from 136 ophthalmic cases and 448 images were included in the analysis. The chatbot answered 299 of multiple-choice questions correctly across all cases (70%). The chatbot’s performance was better on retina questions than neuro-ophthalmology questions (77% vs 58%; difference = 18%; 95% CI, 7.5%-29.4%; χ 2 1 = 11.4; P &lt; .001). The chatbot achieved a better performance on nonimage–based questions compared with image-based questions (82% vs 65%; difference = 17%; 95% CI, 7.8%-25.1%; χ 2 1 = 12.2; P &lt; .001).The chatbot performed best on questions in the retina category (77% correct) and poorest in the neuro-ophthalmology category (58% correct). The chatbot demonstrated intermediate performance on questions from the ocular oncology (72% correct), pediatric ophthalmology (68% correct), uveitis (67% correct), and glaucoma (61% correct) categories. Conclusions and Relevance In this study, the recent version of the chatbot accurately responded to approximately two-thirds of multiple-choice questions pertaining to ophthalmic cases based on imaging interpretation. The multimodal chatbot performed better on questions that did not rely on the interpretation of imaging modalities. As the use of multimodal chatbots becomes increasingly widespread, it is imperative to stress their appropriate integration within medical contexts.

ophthalmology
ChatGPT enters the room: what it means for patient counseling, physician education, academics, and disease management

Bita Momenaei,Hana A. Mansour,Ajay E. Kuriyan,David Xu,Jayanth Sridhar,Daniel S.W. Ting,Yoshihiro Yonekawa

DOI: https://doi.org/10.1097/icu.0000000000001036

2024-02-08

Current Opinion in Ophthalmology

Abstract:Purpose of review This review seeks to provide a summary of the most recent research findings regarding the utilization of ChatGPT, an artificial intelligence (AI)-powered chatbot, in the field of ophthalmology in addition to exploring the limitations and ethical considerations associated with its application. Recent findings ChatGPT has gained widespread recognition and demonstrated potential in enhancing patient and physician education, boosting research productivity, and streamlining administrative tasks. In various studies examining its utility in ophthalmology, ChatGPT has exhibited fair to good accuracy, with its most recent iteration showcasing superior performance in providing ophthalmic recommendations across various ophthalmic disorders such as corneal diseases, orbital disorders, vitreoretinal diseases, uveitis, neuro-ophthalmology, and glaucoma. This proves beneficial for patients in accessing information and aids physicians in triaging as well as formulating differential diagnoses. Despite such benefits, ChatGPT has limitations that require acknowledgment including the potential risk of offering inaccurate or harmful information, dependence on outdated data, the necessity for a high level of education for data comprehension, and concerns regarding patient privacy and ethical considerations within the research domain. Summary ChatGPT is a promising new tool that could contribute to ophthalmic healthcare education and research, potentially reducing work burdens. However, its current limitations necessitate a complementary role with human expert oversight.

ophthalmology
Artificial intelligence chatbot interpretation of ophthalmic multimodal imaging cases

Andrew Mihalache,Ryan S. Huang,Miguel Cruz-Pimentel,Nikhil S. Patil,Marko M. Popovic,Bhadra U. Pandya,Reut Shor,Austin Pereira,Rajeev H. Muni

DOI: https://doi.org/10.1038/s41433-024-03074-5

IF: 4.4563

2024-04-22

Eye

Abstract:The integration of artificial intelligence (AI) within ophthalmology represents a paradigm shift that can redefine patient interactions and healthcare delivery [1]. The large language model ChatGPT-4 (OpenAI) has been recognized for its rapidly advancing proficiency in medical prompt analysis [2, 3]. While earlier iterations of this chatbot were confined to text-based input, our recent work has documented a remarkable expansion in its ophthalmic applications [4,5,6]. In particular, the latest iteration of the chatbot stands at the threshold of potentially revolutionizing the efficiency with which multimodal ophthalmic images are assessed. Ophthalmologists heavily rely on the nuanced interpretation of multimodal imaging, a cornerstone in achieving diagnostic accuracy and improving patient outcomes [7]. Hence, we aimed to evaluate the chatbot's performance on open-ended queries regarding multimodal ophthalmic cases, a crucial step in characterizing the real-world clinical utility of this burgeoning technology.

ophthalmology
The ability of artificial intelligence chatbots ChatGPT and Google Bard to accurately convey pre-operative information for patients undergoing ophthalmological surgeries

Nikhil S Patil,Ryan Huang,Andrew Mihalache,Eli Kisilevsky,Jason Kwok,Marko M Popovic,Georges Nassrallah,Clara Chan,Ashwin Mallipatna,Peter J Kertes,Rajeev H Muni,Nikhil S. Patil,Marko M. Popovic,Peter J. Kertes,Rajeev H. Muni

DOI: https://doi.org/10.1097/iae.0000000000004044

2024-01-09

Retina

Abstract:Introduction: To determine whether the two popular artificial intelligence (AI) chatbots, ChatGPT and Bard, provide high-quality information concerning procedure description, risks, benefits, and alternatives of various ophthalmological surgeries. Methods: ChatGPT and Bard were prompted with questions pertaining to the description, potential risks, benefits, alternatives, and implications of not proceeding with various surgeries in different subspecialties of ophthalmology. Six common ophthalmic procedures were included in our analysis. Two comprehensive ophthalmologists and one sub-specialist graded each response independently using a five-point Likert scale. Results: Likert grading for accuracy was significantly higher for ChatGPT in comparison to Bard (4.5±0.6 vs 3.8±0.8, p<0.0001). Generally, ChatGPT performed better than Bard even when questions were stratified by type of ophthalmological surgery. There was no significant difference between ChatGPT and Bard for response length (2104.7±271.4 characters vs 2441.0±633.9 characters, p=0.12). ChatGPT responded significantly slower than Bard (46.0±3.0 seconds vs 6.6±1.2 seconds, p<0.0001). Conclusions: Both ChatGPT and Bard may offer accessible and high-quality information relevant to the informed consent process for various ophthalmic procedures. Nonetheless, both AI chatbots overlooked probability of adverse events, hence limiting their potential and introducing patients to information that may be difficult to interpret.

ophthalmology
Evaluating the Accuracy of ChatGPT and Google BARD in Fielding Oculoplastic Patient Queries: A Comparative Study on Artificial versus Human Intelligence

Eman M Al-Sharif,Rafaella C Penteado,Nahia Dib El Jalbout,Nicole J Topilow,Marissa K Shoji,Don O Kikkawa,Catherine Y Liu,Bobby S Korn,Eman M. Al-Sharif,Rafaella C. Penteado,Nicole J. Topilow,Marissa K. Shoji,Don O. Kikkawa,Catherine Y. Liu,Bobby S. Korn

DOI: https://doi.org/10.1097/iop.0000000000002567

2024-01-12

Abstract:Purpose: This study evaluates and compares the accuracy of responses from 2 artificial intelligence platforms to patients’ oculoplastics-related questions. Methods: Questions directed toward oculoplastic surgeons were collected, rephrased, and input independently into ChatGPT-3.5 and BARD chatbots, using the prompt: “As an oculoplastic surgeon, how can I respond to my patient’s question?.” Responses were independently evaluated by 4 experienced oculoplastic specialists as comprehensive, correct but inadequate, mixed correct and incorrect/outdated data, and completely incorrect. Additionally, the empathy level, length, and automated readability index of the responses were assessed. Results: A total of 112 patient questions underwent evaluation. The rates of comprehensive, correct but inadequate, mixed, and completely incorrect answers for ChatGPT were 71.4%, 12.9%, 10.5%, and 5.1%, respectively, compared with 53.1%, 18.3%, 18.1%, and 10.5%, respectively, for BARD. ChatGPT showed more empathy (48.9%) than BARD (13.2%). All graders found that ChatGPT outperformed BARD in question categories of postoperative healing, medical eye conditions, and medications. Categorizing questions by anatomy, ChatGPT excelled in answering lacrimal questions (83.8%), while BARD performed best in the eyelid group (60.4%). ChatGPT’s answers were longer and potentially more challenging to comprehend than BARD’s. Conclusion: This study emphasizes the promising role of artificial intelligence-powered chatbots in oculoplastic patient education and support. With continued development, these chatbots may potentially assist physicians and offer patients accurate information, ultimately contributing to improved patient care while alleviating surgeon burnout. However, it is crucial to highlight that artificial intelligence may be good at answering questions, but physician oversight remains essential to ensure the highest standard of care and address complex medical cases.

ophthalmology,surgery
Eyes on AI: ChatGPT's Transformative Potential Impact on Ophthalmology

Jason Dossantos,Jella An,Ramin Javan

DOI: https://doi.org/10.7759/cureus.40765

2023-06-21

Cureus

Abstract:ChatGPT, a large language model by OpenAI, has been adopted in various domains since its release in November 2022, but its application in ophthalmology remains less explored. This editorial assesses ChatGPT's potential applications and limitations in ophthalmology across clinical, educational, and research settings. In clinical settings, ChatGPT can serve as an assistant, offering diagnostic and therapeutic suggestions based on patient data and assisting in patient triage. However, its tendencies to generate inaccurate results and its inability to keep up with recent medical guidelines render it unsuitable for standalone clinical decision-making. Data security and compliance with the Health Insurance Portability and Accountability Act (HIPAA) also pose concerns, given ChatGPT's potential to inadvertently expose sensitive patient information. In education, ChatGPT can generate practice questions, provide explanations, and create patient education materials. However, its performance in answering domain-specific questions is suboptimal. In research, ChatGPT can facilitate literature reviews, data analysis, manuscript development, and peer review, but issues of accuracy, bias, and ethics need careful consideration. Ultimately, ensuring accuracy, ethical integrity, and data privacy is essential when integrating ChatGPT into ophthalmology.

English Else
Performance of ChatGPT in Diagnosis of Corneal Eye Diseases

Mohammad Delsoz,Yeganeh Madadi,Hina Raja,Wuqaas M Munir,Brendan Tamm,Shiva Mehravaran,Mohammad Soleimani,Ali Djalilian,Siamak Yousefi,Wuqaas M. Munir

DOI: https://doi.org/10.1097/ico.0000000000003492

IF: 3.152

2024-02-23

Cornea

Abstract:Purpose: The aim of this study was to assess the capabilities of ChatGPT-4.0 and ChatGPT-3.5 for diagnosing corneal eye diseases based on case reports and compare with human experts. Methods: We randomly selected 20 cases of corneal diseases including corneal infections, dystrophies, and degenerations from a publicly accessible online database from the University of Iowa. We then input the text of each case description into ChatGPT-4.0 and ChatGPT-3.5 and asked for a provisional diagnosis. We finally evaluated the responses based on the correct diagnoses, compared them with the diagnoses made by 3 corneal specialists (human experts), and evaluated interobserver agreements. Results: The provisional diagnosis accuracy based on ChatGPT-4.0 was 85% (17 correct of 20 cases), whereas the accuracy of ChatGPT-3.5 was 60% (12 correct cases of 20). The accuracy of 3 corneal specialists compared with ChatGPT-4.0 and ChatGPT-3.5 was 100% (20 cases, P = 0.23, P = 0.0033), 90% (18 cases, P = 0.99, P = 0.6), and 90% (18 cases, P = 0.99, P = 0.6), respectively. The interobserver agreement between ChatGPT-4.0 and ChatGPT-3.5 was 65% (13 cases), whereas the interobserver agreement between ChatGPT-4.0 and 3 corneal specialists was 85% (17 cases), 80% (16 cases), and 75% (15 cases), respectively. However, the interobserver agreement between ChatGPT-3.5 and each of 3 corneal specialists was 60% (12 cases). Conclusions: The accuracy of ChatGPT-4.0 in diagnosing patients with various corneal conditions was markedly improved than ChatGPT-3.5 and promising for potential clinical integration. A balanced approach that combines artificial intelligence–generated insights with clinical expertise holds a key role for unveiling its full potential in eye care.

ophthalmology
Performance Assessment of an Artificial Intelligence Chatbot in Clinical Vitreoretinal Scenarios

Michael J. Maywood,Ravi Parikh,Avnish Deobhakta,Tedi Begaj

DOI: https://doi.org/10.1097/iae.0000000000004053

2024-01-23

Retina

Abstract:Purpose: To determine how often ChatGPT is able to provide accurate and comprehensive information regarding clinical vitreoretinal scenarios. To assess the types of sources ChatGPT primarily utilizes and to determine if they are hallucinated. Methods: A retrospective cross-sectional study. We designed 40 open-ended clinical scenarios across 4 main topics in vitreoretinal disease. Responses were graded on correctness and comprehensiveness by two blinded retina specialists. The primary outcome was the number of clinical scenarios that ChatGPT answered correctly and comprehensively. Secondary outcomes included: theoretical harm to patients, the distribution of the type of references utilized by the chatbot, and the frequency of hallucinated references. Results: In June 2023, ChatGPT answered 83% (33/40) of clinical scenarios correctly but provided a comprehensive answer in only 52.5% (21/40) of cases. Subgroup analysis demonstrated an average correct score of 86.7% in nAMD, 100% in DR, 76.7% in retinal vascular disease and 70% in the surgical domain. There were 6 incorrect responses with 1 (16.7%) case of no harm, 3 (50%) cases of possible harm and 2 (33.3%) cases of definitive harm. Conclusion: ChatGPT correctly answered more than 80% of complex open-ended vitreoretinal clinical scenarios, with a reduced capability to provide a comprehensive response.

ophthalmology
ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

Michael Balas,Efrem D Mandelcorn,Peng Yan,Edsel B Ing,Sean A Crawford,Parnian Arjmand

DOI: https://doi.org/10.1016/j.jcjo.2024.06.001

2024-08-01

Abstract:Objective: To evaluate the performance of an artificial intelligence (AI) large language model, ChatGPT (version 4.0), for common retinal diseases, in accordance with the American Academy of Ophthalmology (AAO) Preferred Practice Pattern (PPP) guidelines. Design: A cross-sectional survey study design was employed to compare the responses made by ChatGPT to established clinical guidelines. Participants: Responses by the AI were reviewed by a panel of three vitreoretinal specialists for evaluation. Methods: To investigate ChatGPT's comprehension of clinical guidelines, we designed 130 questions covering a broad spectrum of topics within 12 AAO PPP domains of retinal disease These questions were crafted to encompass diagnostic criteria, treatment guidelines, and management strategies, including both medical and surgical aspects of retinal care. A panel of 3 retinal specialists independently evaluated responses on a Likert scale from 1 to 5 based on their relevance, accuracy, and adherence to AAO PPP guidelines. Response readability was evaluated using Flesch Readability Ease and Flesch-Kincaid grade level scores. Results: ChatGPT achieved an overall average score of 4.9/5.0, suggesting high alignment with the AAO PPP guidelines. Scores varied across domains, with the lowest in the surgical management of disease. The responses had a low reading ease score and required a college-to-graduate level of comprehension. Identified errors were related to diagnostic criteria, treatment options, and methodological procedures. Conclusion: ChatGPT 4.0 demonstrated significant potential in generating guideline-concordant responses, particularly for common medical retinal diseases. However, its performance slightly decreased in surgical retina, highlighting the ongoing need for clinician input, further model refinement, and improved comprehensibility.
Evaluating the Artificial Intelligence Performance Growth in Ophthalmic Knowledge

Cheng Jiao,Neel R. Edupuganti,Parth A. Patel,Tommy Bui,Veeral Sheth,Neel Edupuganti,Neel R Edupuganti,Parth A Patel

DOI: https://doi.org/10.7759/cureus.45700

2023-09-21

Cureus

Abstract:OBJECTIVE: We aim to compare the capabilities of Chat Generative Pre-Trained Transformer (ChatGPT)-3.5 and ChatGPT-4.0 (OpenAI, San Francisco, CA, USA) in addressing multiple-choice ophthalmic case challenges.METHODS AND ANALYSIS: Both models' accuracy was compared across different ophthalmology subspecialties using multiple-choice ophthalmic clinical cases provided by the American Academy of Ophthalmology (AAO) "Diagnosis This" questions. Additional analysis was based on image content, question difficulty, character length of models' responses, and model's alignment with responses from human respondents. χ2 test, Fisher's exact test, Student's t-test, and one-way analysis of variance (ANOVA) were conducted where appropriate, with p<0.05 considered significant.RESULTS: GPT-4.0 significantly outperformed GPT-3.5 (75% versus 46%, p<0.01), with the most noticeable improvement in neuro-ophthalmology (100% versus 38%, p=0.03). While both models struggled with uveitis and refractive questions, GPT-4.0 excelled in other areas, such as pediatric questions (82%). In image-related questions, GPT-4.0 also displayed superior accuracy that trended toward significance (73% versus 46%, p=0.07). GPT-4.0 performed better with easier questions (93.8% (least difficult) versus 76.2% (middle) versus 53.3% (most), p=0.03) and generated more concise answers than GPT-3.5 (651.7±342.9 versus 1,112.9±328.8 characters, p<0.01). Moreover, GPT-4.0's answers were more in line with those of AAO respondents (57.3% versus 41.4%, p<0.01), showing a strong correlation between its accuracy and the proportion of AAO respondents who selected GPT-4.0's answer (ρ=0.713, p<0.01).CONCLUSION AND RELEVANCE: Our study demonstrated that GPT-4.0 significantly outperforms GPT-3.5 in addressing ophthalmic case challenges, especially in neuro-ophthalmology, with improved accuracy even in image-related questions. These findings underscore the potential of advancing artificial intelligence (AI) models in enhancing ophthalmic diagnostics and medical education.
Evaluating accuracy and reproducibility of ChatGPT responses to patient-based questions in Ophthalmology: An observational study

Asem A Alqudah,Abdelwahab J Aleshawi,Mohammed Baker,Zaina Alnajjar,Ibrahim Ayasrah,Yaqoot Ta'ani,Mohammad Al Salkhadi,Shaima'a Aljawarneh

DOI: https://doi.org/10.1097/MD.0000000000039120

2024-08-09

Abstract:Chat Generative Pre-Trained Transformer (ChatGPT) is an online large language model that appears to be a popular source of health information, as it can provide patients with answers in the form of human-like text, although the accuracy and safety of its responses are not evident. This study aims to evaluate the accuracy and reproducibility of ChatGPT responses to patients-based questions in ophthalmology. We collected 150 questions from the "Ask an ophthalmologist" page of the American Academy of Ophthalmology, which were reviewed and refined by two ophthalmologists for their eligibility. Each question was inputted into ChatGPT twice using the "new chat" option. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Totally, 117 questions were inputted into ChatGPT, which provided "comprehensive" responses to 70/117 (59.8%) of questions. Concerning reproducibility, it was defined as no difference in grading categories (1 and 2 vs 3 and 4) between the 2 responses for each question. ChatGPT provided reproducible responses to 91.5% of questions. This study shows moderate accuracy and reproducibility of ChatGPT responses to patients' questions in ophthalmology. ChatGPT may be-after more modifications-a supplementary health information source, which should be used as an adjunct, but not a substitute, to medical advice. The reliability of ChatGPT should undergo more investigations.
Assessing the proficiency of artificial intelligence programs in the diagnosis and treatment of cornea, conjunctiva, and eyelid diseases and exploring the advantages of each other benefits

Eyupcan Sensoy,Mehmet Citirik

DOI: https://doi.org/10.1016/j.clae.2024.102125

IF: 3.946

2024-04-01

Contact Lens and Anterior Eye

Abstract:PURPOSE: It was aimed to determine the knowledge level of ChatGPT, Bing, and Bard artificial intelligence programs related to corneal, conjunctival, and eyelid diseases and treatment modalities, to examine their reliability and superiority to each other.METHODS: Forty-one questions related to corneal, conjunctival, and eyelid diseases and treatment modalities were asked to the ChatGPT, Bing, and Bard chatbots. The answers to the questions were compared with the answer keys and grouped as correct or incorrect. Accuracy rates were compared.RESULTS: ChatGPT gave the correct answer to 51.2 % of the questions asked, Bing gave the correct answer to 53.7 %, and Bard gave the correct answer to 68.3 %. There was no significant difference in the rate of correct or incorrect answers to the questions asked for the 3 artificial intelligence chatbots (p = 0.208, Pearson's chi-square test).CONCLUSION: Although information about the cornea, conjunctiva, and eyelid diseases and treatment modalities can be accessed quickly and accurately using up-to-date artificial intelligence programs, the answers may not always be accurate and up-to-date. Care should be taken when evaluating this information.

ophthalmology
Evaluating the potential of ChatGPT-4 in ophthalmology: The good, the bad and the ugly

R K Khanna,J-B Ducloyer,A Hage,A Rezkallah,E Durbant,M Bigoteau,R Mouchel,R Guillon-Rolf,L Le,R Tahiri,J Chammas,C Baudouin

DOI: https://doi.org/10.1016/j.jfo.2023.07.001

Abstract:There is growing interest nowadays for artificial intelligence (AI) in all medical fields. Beyond the direct medical application of AI to medical data, generative AI such as "pre-trained transformer" (GPT) could significantly change the ophthalmology landscape, opening up new avenues for enhancing precision, productivity, and patient outcomes. At present, ChatGPT-4 has been investigated in various ways in ophthalmology for research, medical education, and support for clinical decisions purposes. This article intends to demonstrate the application of ChatGPT-4 within the field of ophthalmology by employing a 'mise en abime' approach. While we explore its potential to enhance the future of ophthalmology care, we will also carefully outline its current limitations and potential risks.
Evaluation of Current Artificial Intelligence Programs on the Knowledge of Glaucoma

Eyupcan Sensoy,Mehmet Citirik

DOI: https://doi.org/10.1055/a-2327-8484

2024-07-26

Klinische Monatsblätter für Augenheilkunde

Abstract:Background To measure the success of three different artificial intelligence chatbots, ChatGPT, Bard, and Bing, in correctly answering questions about glaucoma types and treatment modalities and to examine their superiority over each other. Materials and Methods Thirty-two questions about glaucoma types and treatment modalities were asked using the ChatGPT, Bard, and Bing chatbots. The correct and incorrect answers were also provided. Accuracy rates were compared. Outcomes Questions asked: ChatGPT answered 56.3%, Bard 78.1%, and Bing 59.4% correctly. There was no statistically significant difference between the three artificial intelligence chatbots in the rate of correct and incorrect answers to the questions asked (p = 0.195). Conclusion Artificial intelligence chatbots can be used as a tool to access accurate information regarding glaucoma types and treatment modalities. However, the information obtained is not always accurate, and care should be taken when using this information. Hintergrund Ziel ist es, den Erfolg von drei verschiedenen Chatbots mit künstlicher Intelligenz – ChatGPT, Bard und Bing – bei der richtigen Beantwortung von Fragen zu Glaukomarten und Behandlungsmethoden zu messen und ihre Überlegenheit gegenüber den anderen zu untersuchen. Methoden Mithilfe der Chatbots ChatGPT, Bard und Bing wurden 32 Fragen zu Glaukomarten und Behandlungsmethoden gestellt. Die richtigen und falschen Antworten wurden ebenfalls angegeben. Die Genauigkeitsraten wurden verglichen. Ergebnisse Gestellte Fragen: ChatGPT antwortete 56,3%, Bard 78,1% und Bing 59,4% richtig. Es gab keinen statistisch signifikanten Unterschied in der Rate der richtigen und falschen Antworten auf die gestellten Fragen zwischen den drei Chatbots mit künstlicher Intelligenz (p = 0,195). Schlussfolgerung Chatbots mit künstlicher Intelligenz können als Hilfsmittel eingesetzt werden, um genaue Informationen zu Glaukomarten und Behandlungsmethoden zu erhalten. Die erhaltenen Informationen sind jedoch nicht immer genau und bei der Verwendung dieser Informationen ist Vorsicht geboten. Already Known: Chatbots are new applications that have emerged with the development of artificial intelligence programs. Although the usability of artificial intelligence programs in ophthalmology fields has been tested, the existence and usability of three different artificial intelligence chatbots, which are available for free use, have not been investigated for their superiority over each other in accessing information about glaucoma diseases and treatment methods. Newly described: Although all three artificial intelligence programs were not statistically superior to each other in answering the questions correctly, the more up-to-date Bard and Bing artificial intelligence chatbots answered the questions with higher accuracy rates. Artificial intelligence programs, including current artificial intelligence programs such as Bard and Bing, may encounter various obstacles (paid access, etc.) in accessing current and accurate information. There seems to be a need for further development of artificial intelligence programs and the completion of these deficiencies. Received: 12 August 2023 Accepted: 12 May 2024 Article published online: 24 July 2024 © 2024. Thieme. All rights reserved. Georg Thieme Verlag KG Rüdigerstraße 14, 70469 Stuttgart, Germany

ophthalmology
Performance of Chatgpt in ophthalmology exam; human versus AI

Balci, Ali Safa

DOI: https://doi.org/10.1007/s10792-024-03353-w

IF: 2.029

2024-11-07

International Ophthalmology

Abstract:This cross-sectional study focuses on evaluating the success rate of ChatGPT in answering questions from the 'Resident Training Development Exam' and comparing these results with the performance of the ophthalmology residents.

ophthalmology
Artificial Intelligence in Ophthalmology: A Comparative Analysis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions

Majid Moshirfar,Amal W Altaf,Isabella M Stoakes,Jared J Tuttle,Phillip C Hoopes

DOI: https://doi.org/10.7759/cureus.40822

2023-06-22

Cureus

Abstract:Importance Chat Generative Pre-Trained Transformer (ChatGPT) has shown promising performance in various fields, including medicine, business, and law, but its accuracy in specialty-specific medical questions, particularly in ophthalmology, is still uncertain. Purpose This study evaluates the performance of two ChatGPT models (GPT-3.5 and GPT-4) and human professionals in answering ophthalmology questions from the StatPearls question bank, assessing their outcomes, and providing insights into the integration of artificial intelligence (AI) technology in ophthalmology. Methods ChatGPT's performance was evaluated using 467 ophthalmology questions from the StatPearls question bank. These questions were stratified into 11 subcategories, four difficulty levels, and three generalized anatomical categories. The answer accuracy of GPT-3.5, GPT-4, and human participants was assessed. Statistical analysis was conducted via the Kolmogorov-Smirnov test for normality, one-way analysis of variance (ANOVA) for the statistical significance of GPT-3 versus GPT-4 versus human performance, and repeated unpaired two-sample t-tests to compare the means of two groups. Results GPT-4 outperformed both GPT-3.5 and human professionals on ophthalmology StatPearls questions, except in the "Lens and Cataract" category. The performance differences were statistically significant overall, with GPT-4 achieving higher accuracy (73.2%) compared to GPT-3.5 (55.5%, p-value < 0.001) and humans (58.3%, p-value < 0.001). There were variations in performance across difficulty levels (rated one to four), but GPT-4 consistently performed better than both GPT-3.5 and humans on level-two, -three, and -four questions. On questions of level-four difficulty, human performance significantly exceeded that of GPT-3.5 (p = 0.008). Conclusion The study's findings demonstrate GPT-4's significant performance improvements over GPT-3.5 and human professionals on StatPearls ophthalmology questions. Our results highlight the potential of advanced conversational AI systems to be utilized as important tools in the education and practice of medicine.
Interpretation of Clinical Retinal Images Using an Artificial Intelligence Chatbot

Andrew Mihalache,Ryan S Huang,David Mikhail,Marko M Popovic,Reut Shor,Austin Pereira,Jason Kwok,Peng Yan,David T Wong,Peter J Kertes,Radha P Kohly,Rajeev H Muni

DOI: https://doi.org/10.1016/j.xops.2024.100556

2024-05-23

Abstract:Purpose: To assess the performance of Chat Generative Pre-Trained Transformer-4 in providing accurate diagnoses to retina teaching cases from OCTCases. Design: Cross-sectional study. Subjects: Retina teaching cases from OCTCases. Methods: We prompted a custom chatbot with 69 retina cases containing multimodal ophthalmic images, asking it to provide the most likely diagnosis. In a sensitivity analysis, we inputted increasing amounts of clinical information pertaining to each case until the chatbot achieved a correct diagnosis. We performed multivariable logistic regressions on Stata v17.0 (StataCorp LLC) to investigate associations between the amount of text-based information inputted per prompt and the odds of the chatbot achieving a correct diagnosis, adjusting for the laterality of cases, number of ophthalmic images inputted, and imaging modalities. Main outcome measures: Our primary outcome was the proportion of cases for which the chatbot was able to provide a correct diagnosis. Our secondary outcome was the chatbot's performance in relation to the amount of text-based information accompanying ophthalmic images. Results: Across 69 retina cases collectively containing 139 ophthalmic images, the chatbot was able to provide a definitive, correct diagnosis for 35 (50.7%) cases. The chatbot needed variable amounts of clinical information to achieve a correct diagnosis, where the entire patient description as presented by OCTCases was required for a majority of correctly diagnosed cases (23 of 35 cases, 65.7%). Relative to when the chatbot was only prompted with a patient's age and sex, the chatbot achieved a higher odds of a correct diagnosis when prompted with an entire patient description (odds ratio = 10.1, 95% confidence interval = 3.3-30.3, P < 0.01). Despite providing an incorrect diagnosis for 34 (49.3%) cases, the chatbot listed the correct diagnosis within its differential diagnosis for 7 (20.6%) of these incorrectly answered cases. Conclusions: This custom chatbot was able to accurately diagnose approximately half of the retina cases requiring multimodal input, albeit relying heavily on text-based contextual information that accompanied ophthalmic images. The diagnostic ability of the chatbot in interpretation of multimodal imaging without text-based information is currently limited. The appropriate use of the chatbot in this setting is of utmost importance, given bioethical concerns. Financial disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
Dr. Google vs. Dr. ChatGPT: Exploring the Use of Artificial Intelligence in Ophthalmology by Comparing the Accuracy, Safety, and Readability of Responses to Frequently Asked Patient Questions Regarding Cataracts and Cataract Surgery

Samuel A. Cohen,Arthur Brant,Ann Caroline Fisher,Suzann Pershing,Diana Do,Carolyn Pan

DOI: https://doi.org/10.1080/08820538.2024.2326058

2024-03-24

Seminars in Ophthalmology

Abstract:Purpose Patients are using online search modalities to learn about their eye health. While Google remains the most popular search engine, the use of large language models (LLMs) like ChatGPT has increased. Cataract surgery is the most common surgical procedure in the US, and there is limited data on the quality of online information that populates after searches related to cataract surgery on search engines such as Google and LLM platforms such as ChatGPT. We identified the most common patient frequently asked questions (FAQs) about cataracts and cataract surgery and evaluated the accuracy, safety, and readability of the answers to these questions provided by both Google and ChatGPT. We demonstrated the utility of ChatGPT in writing notes and creating patient education materials.

ophthalmology
The Accuracy of Artificial Intelligence ChatGPT in Oncology Examination Questions

Ronald Chow,Shaakir Hasan,Ajay Zheng,Chenxi Gao,Gilmer Valdes,Francis Yu,Arpit Chhabra,Srinivas Raman,J. Isabelle Choi,Haibo Lin,Charles B. Simone

DOI: https://doi.org/10.1016/j.jacr.2024.07.011

IF: 6.24

2024-08-28

Journal of the American College of Radiology

Abstract:The aim of this study is to assess the accuracy of Chat Generative Pretrained Transformer (ChatGPT) in response to oncology examination questions in the setting of one-shot learning. Consecutive national radiation oncology in-service multiple-choice examinations were collected and inputted into ChatGPT 4o and ChatGPT 3.5 to determine ChatGPT's answers. ChatGPT's answers were then compared with the answer keys to determine whether ChatGPT correctly or incorrectly answered each question and to determine if improvements in responses were seen with the newer ChatGPT version. A total of 600 consecutive questions were inputted into ChatGPT. ChatGPT 4o answered 72.2% questions correctly, whereas 3.5 answered 53.8% questions correctly. There was a significant difference in performance by question category ( P < .01). ChatGPT performed poorer with respect to knowledge of landmark studies and treatment recommendations and planning. ChatGPT is a promising technology, with the latest version showing marked improvement. Although it still has limitations, with further evolution, it may be considered a reliable resource for medical training and decision making in the oncology space.

radiology, nuclear medicine & medical imaging
Are artificial intelligence chatbots a reliable source of information about contact lenses?

Nery García-Porta,Megan Vaughan,Sofia Rendo-González,Ana I Gómez-Varela,Autumn O'Donnell,Joaquim de-Moura,Jorge Novo-Bujan,Marcos Ortega-Hortas,Ana I. Gómez-Varela

DOI: https://doi.org/10.1016/j.clae.2024.102130

IF: 3.946

2024-03-07

Contact Lens and Anterior Eye

Abstract:Introduction Artificial Intelligence (AI) chatbots are able to explain complex concepts using plain language. The aim of this study was to assess the accuracy of three AI chatbots answering common questions related to contact lens (CL) wear. Methods Three open access AI chatbots were compared: Perplexity, Open Assistant and ChatGPT 3.5. Ten general CL questions were asked to all AI chatbots on the same day in two different countries, with the questions asked in Spanish from Spain and in English from the U.K. Two independent optometrists with experience working in each country assessed the accuracy of the answers provided. Also, the AI chatbots' responses were assessed if their outputs showed any bias towards (or against) any eye care professional (ECP). Results The answers obtained by the same AI chatbots were different in Spain and the U.K. Also, statistically significant differences were found between the AI chatbots for accuracy. In the U.K., ChatGPT 3.5 was the most and Open Assistant least accurate (p < 0.01). In Spain, Perplexity and ChatGPT were statistically more accurate than Open Assistant (p < 0.01). All the AI chatbots presented bias, except ChatGPT 3.5 in Spain. Conclusions AI chatbots do not always consider local CL legislation, and their accuracy seems to be dependent on the language used to interact with them. Hence, at this time, although some AI chatbots might be a good source of information for general CL related questions, they cannot replace an ECP.

ophthalmology

Reliability and accuracy of artificial intelligence ChatGPT in providing information on ophthalmic diseases and management to patients

Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images

ChatGPT enters the room: what it means for patient counseling, physician education, academics, and disease management

Artificial intelligence chatbot interpretation of ophthalmic multimodal imaging cases

The ability of artificial intelligence chatbots ChatGPT and Google Bard to accurately convey pre-operative information for patients undergoing ophthalmological surgeries

Evaluating the Accuracy of ChatGPT and Google BARD in Fielding Oculoplastic Patient Queries: A Comparative Study on Artificial versus Human Intelligence

Eyes on AI: ChatGPT's Transformative Potential Impact on Ophthalmology

Performance of ChatGPT in Diagnosis of Corneal Eye Diseases

Performance Assessment of an Artificial Intelligence Chatbot in Clinical Vitreoretinal Scenarios

ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

Evaluating the Artificial Intelligence Performance Growth in Ophthalmic Knowledge

Evaluating accuracy and reproducibility of ChatGPT responses to patient-based questions in Ophthalmology: An observational study

Assessing the proficiency of artificial intelligence programs in the diagnosis and treatment of cornea, conjunctiva, and eyelid diseases and exploring the advantages of each other benefits

Evaluating the potential of ChatGPT-4 in ophthalmology: The good, the bad and the ugly

Evaluation of Current Artificial Intelligence Programs on the Knowledge of Glaucoma

Performance of Chatgpt in ophthalmology exam; human versus AI

Artificial Intelligence in Ophthalmology: A Comparative Analysis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions

Interpretation of Clinical Retinal Images Using an Artificial Intelligence Chatbot

Dr. Google vs. Dr. ChatGPT: Exploring the Use of Artificial Intelligence in Ophthalmology by Comparing the Accuracy, Safety, and Readability of Responses to Frequently Asked Patient Questions Regarding Cataracts and Cataract Surgery

The Accuracy of Artificial Intelligence ChatGPT in Oncology Examination Questions

Are artificial intelligence chatbots a reliable source of information about contact lenses?