Bystander effects and radiotherapy.

A. Marín,Margarita Martin,O. Liñán,F. Alvarenga,Mario López,L. Fernández,D. Büchser,L. Cerezo

DOI: https://doi.org/10.1016/j.rpor.2014.08.004

Reports of Practical Oncology & Radiotherapy

Abstract:

What problem does this paper attempt to address?

Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images

Andrew Mihalache,Ryan S. Huang,Marko M. Popovic,Nikhil S. Patil,Bhadra U. Pandya,Reut Shor,Austin Pereira,Jason M. Kwok,Peng Yan,David T. Wong,Peter J. Kertes,Rajeev H. Muni

DOI: https://doi.org/10.1001/jamaophthalmol.2024.0017

IF: 8.253

2024-02-29

JAMA Ophthalmology

Abstract:Importance Ophthalmology is reliant on effective interpretation of multimodal imaging to ensure diagnostic accuracy. The new ability of ChatGPT-4 (OpenAI) to interpret ophthalmic images has not yet been explored. Objective To evaluate the performance of the novel release of an artificial intelligence chatbot that is capable of processing imaging data. Design, Setting, and Participants This cross-sectional study used a publicly available dataset of ophthalmic cases from OCTCases, a medical education platform based out of the Department of Ophthalmology and Vision Sciences at the University of Toronto, with accompanying clinical multimodal imaging and multiple-choice questions. Across 137 available cases, 136 contained multiple-choice questions (99%). Exposures The chatbot answered questions requiring multimodal input from October 16 to October 23, 2023. Main Outcomes and Measures The primary outcome was the accuracy of the chatbot in answering multiple-choice questions pertaining to image recognition in ophthalmic cases, measured as the proportion of correct responses. χ 2 Tests were conducted to compare the proportion of correct responses across different ophthalmic subspecialties. Results A total of 429 multiple-choice questions from 136 ophthalmic cases and 448 images were included in the analysis. The chatbot answered 299 of multiple-choice questions correctly across all cases (70%). The chatbot’s performance was better on retina questions than neuro-ophthalmology questions (77% vs 58%; difference = 18%; 95% CI, 7.5%-29.4%; χ 2 1 = 11.4; P &lt; .001). The chatbot achieved a better performance on nonimage–based questions compared with image-based questions (82% vs 65%; difference = 17%; 95% CI, 7.8%-25.1%; χ 2 1 = 12.2; P &lt; .001).The chatbot performed best on questions in the retina category (77% correct) and poorest in the neuro-ophthalmology category (58% correct). The chatbot demonstrated intermediate performance on questions from the ocular oncology (72% correct), pediatric ophthalmology (68% correct), uveitis (67% correct), and glaucoma (61% correct) categories. Conclusions and Relevance In this study, the recent version of the chatbot accurately responded to approximately two-thirds of multiple-choice questions pertaining to ophthalmic cases based on imaging interpretation. The multimodal chatbot performed better on questions that did not rely on the interpretation of imaging modalities. As the use of multimodal chatbots becomes increasingly widespread, it is imperative to stress their appropriate integration within medical contexts.

ophthalmology
Artificial intelligence chatbot interpretation of ophthalmic multimodal imaging cases

Andrew Mihalache,Ryan S. Huang,Miguel Cruz-Pimentel,Nikhil S. Patil,Marko M. Popovic,Bhadra U. Pandya,Reut Shor,Austin Pereira,Rajeev H. Muni

DOI: https://doi.org/10.1038/s41433-024-03074-5

IF: 4.4563

2024-04-22

Eye

Abstract:The integration of artificial intelligence (AI) within ophthalmology represents a paradigm shift that can redefine patient interactions and healthcare delivery [1]. The large language model ChatGPT-4 (OpenAI) has been recognized for its rapidly advancing proficiency in medical prompt analysis [2, 3]. While earlier iterations of this chatbot were confined to text-based input, our recent work has documented a remarkable expansion in its ophthalmic applications [4,5,6]. In particular, the latest iteration of the chatbot stands at the threshold of potentially revolutionizing the efficiency with which multimodal ophthalmic images are assessed. Ophthalmologists heavily rely on the nuanced interpretation of multimodal imaging, a cornerstone in achieving diagnostic accuracy and improving patient outcomes [7]. Hence, we aimed to evaluate the chatbot's performance on open-ended queries regarding multimodal ophthalmic cases, a crucial step in characterizing the real-world clinical utility of this burgeoning technology.

ophthalmology
APPLICATIONS OF MULTIMODAL GENERATIVE ARTIFICIAL INTELLIGENCE IN A REAL-WORLD RETINA CLINIC SETTING

Seyyedehfatemeh Ghalibafan,David J Taylor Gonzalez,Louis Z Cai,Brandon Graham Chou,Sugi Panneerselvam,Spencer Conrad Barrett,Mak B Djulbegovic,Nicolas A Yannuzzi

DOI: https://doi.org/10.1097/IAE.0000000000004204

2024-10-01

Retina

Abstract:Purpose: This study evaluates a large language model, Generative Pre-trained Transformer 4 with vision, for diagnosing vitreoretinal diseases in real-world ophthalmology settings. Methods: A retrospective cross-sectional study at Bascom Palmer Eye Clinic, analyzing patient data from January 2010 to March 2023, assesses Generative Pre-trained Transformer 4 with vision's performance on retinal image analysis and International Classification of Diseases 10th revision coding across 2 patient groups: simpler cases (Group A) and complex cases (Group B) requiring more in-depth analysis. Diagnostic accuracy was assessed through open-ended questions and multiple-choice questions independently verified by three retina specialists. Results: In 256 eyes from 143 patients, Generative Pre-trained Transformer 4-V demonstrated a 13.7% accuracy for open-ended questions and 31.3% for multiple-choice questions, with International Classification of Diseases 10th revision code accuracies at 5.5% and 31.3%, respectively. Accurately diagnosed posterior vitreous detachment, nonexudative age-related macular degeneration, and retinal detachment. International Classification of Diseases 10th revision coding was most accurate for nonexudative age-related macular degeneration, central retinal vein occlusion, and macular hole in OEQs, and for posterior vitreous detachment, nonexudative age-related macular degeneration, and retinal detachment in multiple-choice questions. No significant difference in diagnostic or coding accuracy was found in Groups A and B. Conclusion: Generative Pre-trained Transformer 4 with vision has potential in clinical care and record keeping, particularly with standardized questions. Its effectiveness in open-ended scenarios is limited, indicating a significant limitation in providing complex medical advice.
Uncovering Language Disparity of ChatGPT in Healthcare: Non-English Clinical Environment for Retinal Vascular Disease Classification

Xiaocong Liu,Jiageng Wu,An Shao,Wenyue Shen,Panpan Ye,Yao Wang,Juan Ye,Kai Jin,Jie Yang

DOI: https://doi.org/10.1101/2023.06.28.23291931

2023-01-01

Abstract:Objective To evaluate the effectiveness and reasoning ability of ChatGPT in diagnosing retinal vascular diseases in the Chinese clinical environment. Materials and Methods We collected 1226 fundus fluorescein angiography reports and corresponding diagnosis written in Chinese, and tested ChatGPT with four prompting strategies (direct diagnosis or diagnosis with explanation and in Chinese or English). Results ChatGPT using English prompt for direct diagnosis achieved the best performance, with F1-score of 80.05%, which was inferior to ophthalmologists (89.35%) but close to ophthalmologist interns (82.69%). Although ChatGPT can derive reasoning process with a low error rate, mistakes such as misinformation (1.96%), and hallucination (0.59%) still exist. Discussion and Conclusions ChatGPT can serve as a helpful medical assistant to provide diagnosis under non-English clinical environments, but there are still performance gaps, language disparity, and errors compared to professionals, which demonstrates the potential limitations and the desiration to continually explore more robust LLMs in ophthalmology practice. ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement The work is supported by Natural Science Foundation of China (grant number: 82201195). ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Ethics committee/IRB of Second Affiliated Hospital, School of Medicine, Zhejiang University gave ethical approval for this work.(IRB:[NCT04718532][1]) I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data will be made available for research purposes upon request. Data requests are to be directed to jinkai{at}zju.edu.cn. [1]: /lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT04718532&atom=%2Fmedrxiv%2Fearly%2F2023%2F07%2F14%2F2023.06.28.23291931.atom
Performance Assessment of an Artificial Intelligence Chatbot in Clinical Vitreoretinal Scenarios

Michael J. Maywood,Ravi Parikh,Avnish Deobhakta,Tedi Begaj

DOI: https://doi.org/10.1097/iae.0000000000004053

2024-01-23

Retina

Abstract:Purpose: To determine how often ChatGPT is able to provide accurate and comprehensive information regarding clinical vitreoretinal scenarios. To assess the types of sources ChatGPT primarily utilizes and to determine if they are hallucinated. Methods: A retrospective cross-sectional study. We designed 40 open-ended clinical scenarios across 4 main topics in vitreoretinal disease. Responses were graded on correctness and comprehensiveness by two blinded retina specialists. The primary outcome was the number of clinical scenarios that ChatGPT answered correctly and comprehensively. Secondary outcomes included: theoretical harm to patients, the distribution of the type of references utilized by the chatbot, and the frequency of hallucinated references. Results: In June 2023, ChatGPT answered 83% (33/40) of clinical scenarios correctly but provided a comprehensive answer in only 52.5% (21/40) of cases. Subgroup analysis demonstrated an average correct score of 86.7% in nAMD, 100% in DR, 76.7% in retinal vascular disease and 70% in the surgical domain. There were 6 incorrect responses with 1 (16.7%) case of no harm, 3 (50%) cases of possible harm and 2 (33.3%) cases of definitive harm. Conclusion: ChatGPT correctly answered more than 80% of complex open-ended vitreoretinal clinical scenarios, with a reduced capability to provide a comprehensive response.

ophthalmology
Diagnosing Glaucoma Based on the Ocular Hypertension Treatment Study Dataset Using Chat Generative Pre-Trained Transformer as a Large Language Model

Hina Raja,Xiaoqin Huang,Mohammad Delsoz,Yeganeh Madadi,Asma Poursoroush,Asim Munawar,Malik Y Kahook,Siamak Yousefi

DOI: https://doi.org/10.1016/j.xops.2024.100599

2024-08-22

Abstract:Purpose: To evaluate the capabilities of Chat Generative Pre-Trained Transformer (ChatGPT), as a large language model (LLM), for diagnosing glaucoma using the Ocular Hypertension Treatment Study (OHTS) dataset, and comparing the diagnostic capability of ChatGPT 3.5 and ChatGPT 4.0. Design: Prospective data collection study. Participants: A total of 3170 eyes of 1585 subjects from the OHTS were included in this study. Methods: We selected demographic, clinical, ocular, visual field, optic nerve head photo, and history of disease parameters of each participant and developed case reports by converting tabular data into textual format based on information from both eyes of all subjects. We then developed a procedure using the application programming interface of ChatGPT, a LLM-based chatbot, to automatically input prompts into a chat box. This was followed by querying 2 different generations of ChatGPT (versions 3.5 and 4.0) regarding the underlying diagnosis of each subject. We then evaluated the output responses based on several objective metrics. Main outcome measures: Area under the receiver operating characteristic curve (AUC), accuracy, specificity, sensitivity, and F1 score. Results: Chat Generative Pre-Trained Transformer 3.5 achieved AUC of 0.74, accuracy of 66%, specificity of 64%, sensitivity of 85%, and F1 score of 0.72. Chat Generative Pre-Trained Transformer 4.0 obtained AUC of 0.76, accuracy of 87%, specificity of 90%, sensitivity of 61%, and F1 score of 0.92. Conclusions: The accuracy of ChatGPT 4.0 in diagnosing glaucoma based on input data from OHTS was promising. The overall accuracy of ChatGPT 4.0 was higher than ChatGPT 3.5. However, ChatGPT 3.5 was found to be more sensitive than ChatGPT 4.0. In its current forms, ChatGPT may serve as a useful tool in exploring disease status of ocular hypertensive eyes when specific data are available for analysis. In the future, leveraging LLMs with multimodal capabilities, allowing for integration of imaging and diagnostic testing as part of the analyses, could further enhance diagnostic capabilities and enhance diagnostic accuracy. Financial disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
Uncovering Language Disparity of ChatGPT in Healthcare: Non-English Clinical Environment for Retinal Vascular Disease Classification (Preprint)

Xiaocong Liu,Jiageng Wu,Anwen Shao,Wei Shen,Panpan Ye,Yao Wang,Juan Ye,Kai Jin,Jie Yang

DOI: https://doi.org/10.2196/preprints.51926

2023-01-01

Abstract:Benefiting from the exceptional ability of text understanding and rich knowledge, large language models (LLMs) like ChatGPT, have shown great potential in English clinical environments. However, the performance of ChatGPT in non-English clinical settings, as well as its reasoning, have not been explored in-depth. To evaluate ChatGPT’s diagnostic performance and inference abilities for retinal vascular diseases in a non-English clinical environment. In this cross-sectional study, we collected 1226 fundus fluorescein angiography reports and corresponding diagnosis written in Chinese, and tested ChatGPT with four prompting strategies (direct diagnosis or diagnosis with explanation and in Chinese or English). ChatGPT using English prompt for direct diagnosis achieved the best performance, with F1-score of 80.05%, which was inferior to ophthalmologists (89.35%) but close to ophthalmologist interns (82.69%). Although ChatGPT can derive reasoning process with a low error rate, mistakes such as misinformation (1.96%), and hallucination (0.59%) still exist. ChatGPT can serve as a helpful medical assistant to provide diagnosis under non-English clinical environments, but there are still performance gaps, language disparity, and errors compared to professionals, which demonstrates the potential limitations and the desiration to continually explore more robust LLMs in ophthalmology practice. ClinicalTrials.gov NCT04718532
ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

Michael Balas,Efrem D Mandelcorn,Peng Yan,Edsel B Ing,Sean A Crawford,Parnian Arjmand

DOI: https://doi.org/10.1016/j.jcjo.2024.06.001

2024-08-01

Abstract:Objective: To evaluate the performance of an artificial intelligence (AI) large language model, ChatGPT (version 4.0), for common retinal diseases, in accordance with the American Academy of Ophthalmology (AAO) Preferred Practice Pattern (PPP) guidelines. Design: A cross-sectional survey study design was employed to compare the responses made by ChatGPT to established clinical guidelines. Participants: Responses by the AI were reviewed by a panel of three vitreoretinal specialists for evaluation. Methods: To investigate ChatGPT's comprehension of clinical guidelines, we designed 130 questions covering a broad spectrum of topics within 12 AAO PPP domains of retinal disease These questions were crafted to encompass diagnostic criteria, treatment guidelines, and management strategies, including both medical and surgical aspects of retinal care. A panel of 3 retinal specialists independently evaluated responses on a Likert scale from 1 to 5 based on their relevance, accuracy, and adherence to AAO PPP guidelines. Response readability was evaluated using Flesch Readability Ease and Flesch-Kincaid grade level scores. Results: ChatGPT achieved an overall average score of 4.9/5.0, suggesting high alignment with the AAO PPP guidelines. Scores varied across domains, with the lowest in the surgical management of disease. The responses had a low reading ease score and required a college-to-graduate level of comprehension. Identified errors were related to diagnostic criteria, treatment options, and methodological procedures. Conclusion: ChatGPT 4.0 demonstrated significant potential in generating guideline-concordant responses, particularly for common medical retinal diseases. However, its performance slightly decreased in surgical retina, highlighting the ongoing need for clinician input, further model refinement, and improved comprehensibility.
A User-friendly Approach for the Diagnosis of Diabetic Retinopathy Using ChatGPT and Automated Machine Learning

S Saeed Mohammadi,Quan Dong Nguyen

DOI: https://doi.org/10.1016/j.xops.2024.100495

2024-02-21

Abstract:Purpose: To assess the capabilities of Chat Generative Pre-trained Transformer (ChatGPT) and Vertex AI in executing code-free preprocessing, training machine learning (ML) models, and analyzing the data. Design: Evaluation of diagnostic test or technology. Participants: ChatGPT and Vetrex AI as publicly available large language model and ML platform, respectively. Methods: ChatGPT was employed to improve the resolution of fundus photography images from the Methods to Evaluate Segmentation and Indexing Techniques in the field of Retinal Ophthalmology (Messidor-2) open-source dataset using the Contrast Limited Adaptive Histogram Equalization (CLAHE) technique by Fiji software. Subsequently, Vertex AI, an automated ML (AutoML) platform, was utilized to develop 2 classification models. The first model served as a binary classifier for detecting the presence of diabetic retinopathy (DR), while the second determined its severity. Finally, ChatGPT was used to provide scripts for R and Python programming languages for data analysis and was also directly employed in analyzing the data in a code-free method. Main outcome measures: Evaluating the utility of ChatGPT in generating scripts for preprocessing images using Fiji and analyzing data across Python and R and assessing its potential in analyzing data through a code-free method. Investigating the capabilities of Vertex AI to train image classification models for detection of DR and its severity. Results: Two ML models were trained using 1740 images from the Messidor-2 database. The first model, designed to detect the severity of DR, achieved an area under the precision-recall curve (AUPRC) of 0.81, with a precision rate of 81.81% and recall of 72.83%. The second model, tailored for the detection of the presence of DR, recorded a precision and recall of 84.48% with an AUPRC of 0.90. Conclusions: ChatGPT and Vertex AI have the potential to enable physicians without coding expertise to preprocess images, analyze data, and train ML models. Financial disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
Applications of ChatGPT in the diagnosis, management, education, and research of retinal diseases: a scoping review

Victor C F Bellanda,Mateus Lins Dos Santos,Daniel Araujo Ferraz,Rodrigo Jorge,Gustavo Barreto Melo

DOI: https://doi.org/10.1186/s40942-024-00595-9

2024-10-17

Abstract:Purpose: This scoping review aims to explore the current applications of ChatGPT in the retina field, highlighting its potential, challenges, and limitations. Methods: A comprehensive literature search was conducted across multiple databases, including PubMed, Scopus, MEDLINE, and Embase, to identify relevant articles published from 2022 onwards. The inclusion criteria focused on studies evaluating the use of ChatGPT in retinal healthcare. Data were extracted and synthesized to map the scope of ChatGPT's applications in retinal care, categorizing articles into various practical application areas such as academic research, charting, coding, diagnosis, disease management, and patient counseling. Results: A total of 68 articles were included in the review, distributed across several categories: 8 related to academics and research, 5 to charting, 1 to coding and billing, 44 to diagnosis, 49 to disease management, 2 to literature consulting, 23 to medical education, and 33 to patient counseling. Many articles were classified into multiple categories due to overlapping topics. The findings indicate that while ChatGPT shows significant promise in areas such as medical education and diagnostic support, concerns regarding accuracy, reliability, and the potential for misinformation remain prevalent. Conclusion: ChatGPT offers substantial potential in advancing retinal healthcare by supporting clinical decision-making, enhancing patient education, and automating administrative tasks. However, its current limitations, particularly in clinical accuracy and the risk of generating misinformation, necessitate cautious integration into practice, with continuous oversight from healthcare professionals. Future developments should focus on improving accuracy, incorporating up-to-date medical guidelines, and minimizing the risks associated with AI-driven healthcare tools.
ChatGPT Assisting Diagnosis of Neuro-ophthalmology Diseases Based on Case Reports

Yeganeh Madadi,Mohammad Delsoz,Priscilla A. Lao,Joseph W. Fong,TJ Hollingsworth,Malik Y. Kahook,Siamak Yousefi

2023-09-05

Abstract:Objective: To evaluate the efficiency of large language models (LLMs) such as ChatGPT to assist in diagnosing neuro-ophthalmic diseases based on detailed case descriptions. Methods: We selected 22 different case reports of neuro-ophthalmic diseases from a publicly available online database. These cases included a wide range of chronic and acute diseases that are commonly seen by neuro-ophthalmic sub-specialists. We inserted the text from each case as a new prompt into both ChatGPT v3.5 and ChatGPT Plus v4.0 and asked for the most probable diagnosis. We then presented the exact information to two neuro-ophthalmologists and recorded their diagnoses followed by comparison to responses from both versions of ChatGPT. Results: ChatGPT v3.5, ChatGPT Plus v4.0, and the two neuro-ophthalmologists were correct in 13 (59%), 18 (82%), 19 (86%), and 19 (86%) out of 22 cases, respectively. The agreement between the various diagnostic sources were as follows: ChatGPT v3.5 and ChatGPT Plus v4.0, 13 (59%); ChatGPT v3.5 and the first neuro-ophthalmologist, 12 (55%); ChatGPT v3.5 and the second neuro-ophthalmologist, 12 (55%); ChatGPT Plus v4.0 and the first neuro-ophthalmologist, 17 (77%); ChatGPT Plus v4.0 and the second neuro-ophthalmologist, 16 (73%); and first and second neuro-ophthalmologists 17 (17%). Conclusions: The accuracy of ChatGPT v3.5 and ChatGPT Plus v4.0 in diagnosing patients with neuro-ophthalmic diseases was 59% and 82%, respectively. With further development, ChatGPT Plus v4.0 may have potential to be used in clinical care settings to assist clinicians in providing quick, accurate diagnoses of patients in neuro-ophthalmology. The applicability of using LLMs like ChatGPT in clinical settings that lack access to subspeciality trained neuro-ophthalmologists deserves further research.

Computers and Society,Artificial Intelligence,Computation and Language
Correlations between parameters of the social network and treatment outcomes of people suffering from schizophrenia seven years after the first hospitalization *

A. Cechnicki,A. Wojciechowska

Abstract:aim. The study investigated correlations between selected parameters of social networks of 64 patients with schizophrenia (DSM III), and the aims of treatment such as: motivation to receive treatment, insight, compliance in taking medication, satisfaction with treatment, and treatment outcomes in the area of clinical, social and family functioning seven years after the first hospital admission. Material and method. The parameters of social networks were checked with Bizoń’s questionnaire. Treatment outcomes criteria were assessed with use of competent judges’ method. The intensity of psychopathological symptoms was measured on the BPRS-LA scale. results. A compound system of social support and a large social network, with a high level of support, correlate in a beneficial way with a higher subjective satisfaction with the treatment. A large extra-familial network, with a high level of support, correlates with a better insight into the illness. The larger somebody’s social network, including the extra-familial network and the high level of incoming support, the fewer positive and negative symptoms they had, and the milder the course of their illness. A larger range of the network correlates with a smaller number of relapses and the total hospitalisation time. Those who have a large network that provides a high level of support, both in the family and outside it, have been less frequently hospitalised in outpatient care. No connection was found between the network’s parameters and the number of inpatient hospitalisations. Conclusions. People with a large network, including the extra-familial network, who receive a high level of social support, function better in society, do not become regressive in their professional lives and there are fewer burdens in their family life. A high level of social support correlates with better family functioning. Families of people with schizophrenia who have a large extra-familial network with a high level of support experience less deterioration and disintegration, less criticism and rejection.
Evaluating the Artificial Intelligence Performance Growth in Ophthalmic Knowledge

Cheng Jiao,Neel R. Edupuganti,Parth A. Patel,Tommy Bui,Veeral Sheth,Neel Edupuganti,Neel R Edupuganti,Parth A Patel

DOI: https://doi.org/10.7759/cureus.45700

2023-09-21

Cureus

Abstract:OBJECTIVE: We aim to compare the capabilities of Chat Generative Pre-Trained Transformer (ChatGPT)-3.5 and ChatGPT-4.0 (OpenAI, San Francisco, CA, USA) in addressing multiple-choice ophthalmic case challenges.METHODS AND ANALYSIS: Both models' accuracy was compared across different ophthalmology subspecialties using multiple-choice ophthalmic clinical cases provided by the American Academy of Ophthalmology (AAO) "Diagnosis This" questions. Additional analysis was based on image content, question difficulty, character length of models' responses, and model's alignment with responses from human respondents. χ2 test, Fisher's exact test, Student's t-test, and one-way analysis of variance (ANOVA) were conducted where appropriate, with p<0.05 considered significant.RESULTS: GPT-4.0 significantly outperformed GPT-3.5 (75% versus 46%, p<0.01), with the most noticeable improvement in neuro-ophthalmology (100% versus 38%, p=0.03). While both models struggled with uveitis and refractive questions, GPT-4.0 excelled in other areas, such as pediatric questions (82%). In image-related questions, GPT-4.0 also displayed superior accuracy that trended toward significance (73% versus 46%, p=0.07). GPT-4.0 performed better with easier questions (93.8% (least difficult) versus 76.2% (middle) versus 53.3% (most), p=0.03) and generated more concise answers than GPT-3.5 (651.7±342.9 versus 1,112.9±328.8 characters, p<0.01). Moreover, GPT-4.0's answers were more in line with those of AAO respondents (57.3% versus 41.4%, p<0.01), showing a strong correlation between its accuracy and the proportion of AAO respondents who selected GPT-4.0's answer (ρ=0.713, p<0.01).CONCLUSION AND RELEVANCE: Our study demonstrated that GPT-4.0 significantly outperforms GPT-3.5 in addressing ophthalmic case challenges, especially in neuro-ophthalmology, with improved accuracy even in image-related questions. These findings underscore the potential of advancing artificial intelligence (AI) models in enhancing ophthalmic diagnostics and medical education.
Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis

Pusheng Xu,Xiaolan Chen,Ziwei Zhao,Danli Shi

DOI: https://doi.org/10.1136/bjo-2023-325054

2024-09-20

British Journal of Ophthalmology

Abstract:Purpose To evaluate the capabilities and incapabilities of a GPT-4V(ision)-based chatbot in interpreting ocular multimodal images. Methods We developed a digital ophthalmologist app using GPT-4V and evaluated its performance with a dataset (60 images, 60 ophthalmic conditions, 6 modalities) that included slit-lamp, scanning laser ophthalmoscopy, fundus photography of the posterior pole (FPP), optical coherence tomography, fundus fluorescein angiography and ocular ultrasound images. The chatbot was tested with ten open-ended questions per image, covering examination identification, lesion detection, diagnosis and decision support. The responses were manually assessed for accuracy, usability, safety and diagnosis repeatability. Auto-evaluation was performed using sentence similarity and GPT-4-based auto-evaluation. Results Out of 600 responses, 30.6% were accurate, 21.5% were highly usable and 55.6% were deemed as no harm. GPT-4V performed best with slit-lamp images, with 42.0%, 38.5% and 68.5% of the responses being accurate, highly usable and no harm, respectively. However, its performance was weaker in FPP images, with only 13.7%, 3.7% and 38.5% in the same categories. GPT-4V correctly identified 95.6% of the imaging modalities and showed varying accuracies in lesion identification (25.6%), diagnosis (16.1%) and decision support (24.0%). The overall repeatability of GPT-4V in diagnosing ocular images was 63.3% (38/60). The overall sentence similarity between responses generated by GPT-4V and human answers is 55.5%, with Spearman correlations of 0.569 for accuracy and 0.576 for usability. Conclusion GPT-4V currently is not yet suitable for clinical decision-making in ophthalmology. Our study serves as a benchmark for enhancing ophthalmic multimodal models.

ophthalmology
Development and Evaluation of Aeyeconsult: A Novel Ophthalmology Chatbot Leveraging Verified Textbook Knowledge and GPT-4

Maxwell B Singer,Julia J Fu,Jessica Chow,Christopher C Teng

DOI: https://doi.org/10.1016/j.jsurg.2023.11.019

Abstract:Objective: There has been much excitement on the use of large language models (LLMs) such as ChatGPT in ophthalmology. However, LLMs are limited in that they are trained on unverified information and do not cite their sources. This paper highlights a new methodology to create a generative AI chatbot to answer eye care related questions which uses only verified ophthalmology textbooks as data and cites its sources. Setting: Yale School of Medicine Department of Ophthalmology and Visual Science. Design/methods: Aeyeconsult, an ophthalmology chatbot, was developed using GPT-4 (the LLM used to power the publicly available chatbot ChatGPT-4), LangChain, and Pinecone. Ophthalmology textbooks were processed into embeddings and stored in Pinecone. User queries were similarly converted, compared to stored embeddings, and GPT-4 generated responses. The interface was adapted from public code. Both Aeyeconsult and ChatGPT-4 were tested on the same 260 questions from OphthoQuestions.com, with the first response from Aeyeconsult and ChatGPT-4 recorded as the answer. Results: Aeyeconsult outperformed ChatGPT-4 on the OKAP dataset, with 83.4% correct answers compared to 69.2% (p = 0.0118). Aeyeconsult also had fewer instances of no answer and multiple answers. Both systems performed best in General Medicine, with Aeyeconsult achieving 96.2% accuracy. Aeyeconsult's weakest performance was in Clinical Optics at 68.1%, but it still outperformed ChatGPT-4 in this category (45.5%). Conclusion: LLMs may be useful in answering ophthalmology questions but their trustworthiness and accuracy is limited due to training on unverified internet data and lack of source citation. We used a new methodology, using verified ophthalmology textbooks as source material and providing citations, to mitigate these issues, resulting in a chatbot more accurate than ChatGPT-4 in answering OKAPs style questions.
Precision of artificial intelligence in paediatric cardiology multimodal image interpretation

Michael N. Gritti,Rahil Prajapati,Dolev Yissar,Conall T. Morgan

DOI: https://doi.org/10.1017/s1047951124036035

2024-11-13

Cardiology in the Young

Abstract:Multimodal imaging is crucial for diagnosis and treatment in paediatric cardiology. However, the proficiency of artificial intelligence chatbots, like ChatGPT-4, in interpreting these images has not been assessed. This cross-sectional study evaluates the precision of ChatGPT-4 in interpreting multimodal images for paediatric cardiology knowledge assessment, including echocardiograms, angiograms, X-rays, and electrocardiograms. One hundred multiple-choice questions with accompanying images from the textbook Pediatric Cardiology Board Review were randomly selected. The chatbot was prompted to answer these questions with and without the accompanying images. Statistical analysis was done using X 2 , Fisher's exact, and McNemar tests. Results showed that ChatGPT-4 answered 41% of questions with images correctly, performing best on those with electrocardiograms (54%) and worst on those with angiograms (29%). Without the images, ChatGPT-4's performance was similar at 37% (difference = 4%, 95% confidence interval (CI) –9.4% to 17.2%, p = 0.56). The chatbot performed significantly better when provided the image of an electrocardiogram than without (difference = 18, 95% CI 4.0% to 31.9%, p < 0.04). In cases of incorrect answers, ChatGPT-4 was more inconsistent with an image than without (difference = 21%, 95% CI 3.5% to 36.9%, p < 0.02). In conclusion, ChatGPT-4 performed poorly in answering image-based multiple-choice questions in paediatric cardiology. Its accuracy in answering questions with images was similar to without, indicating limited multimodal image interpretation capabilities. Substantial training is required before clinical integration can be considered. Further research is needed to assess the clinical reasoning skills and progression of ChatGPT in paediatric cardiology for clinical and academic utility.

pediatrics,cardiac & cardiovascular systems
Conversational AI Models for ophthalmic diagnosis: Comparison of ChatGPT and the Isabel Pro Differential Diagnosis Generator

Michael Balas,Edsel B. Ing

DOI: https://doi.org/10.1016/j.jfop.2023.100005

2023-03-01

JFO Open Ophthalmology

Abstract:Background With the rapidly growing field of conversational artificial intelligence (AI), it is becoming increasingly likely that these technologies will revolutionize the way physicians diagnose and treat patients. The purpose of this study is to evaluate the use of conversational AI language models, specifically ChatGPT, for the diagnosis of ophthalmic disease, and to compare it to existing tools, namely the Isabel Pro Differential Diagnosis Generator. Methods Prospective, comparative evaluation of ChatGPT and Isabel in formulating provisional and differential diagnoses from a set of rich text case report descriptions. Ten ophthalmology patient cases were selected at random from a publicly available online database of ophthalmic case reports. The text details of each case were input into ChatGPT and Isabel. Their ability to identify the actual diagnosis and provide relevant differential diagnoses was compared. Results ChatGPT identified the correct diagnosis in 9/10 cases while having the correct diagnosis listed in all 10/10 of its lists of differentials. Isabel identified only 1/10 provisional diagnoses correctly, however it included the correct diagnosis in 7/10 of its differential diagnosis lists. The median position of the correct diagnosis in the ranked differential lists was 1.0 (IQR 1.0 to 2.8) for ChatGPT versus 5.5 (IQR 3.3 to 10.0) for Isabel. Conclusions Conversational AI models such as ChatGPT have potential value in the diagnosis of ophthalmic conditions, particularly for primary-care providers. As further iterations of models are deployed, additional studies investigating their capabilities are needed to determine the best ways to integrate them into practice.
Performance of ChatGPT in Diagnosis of Corneal Eye Diseases

Mohammad Delsoz,Yeganeh Madadi,Hina Raja,Wuqaas M Munir,Brendan Tamm,Shiva Mehravaran,Mohammad Soleimani,Ali Djalilian,Siamak Yousefi,Wuqaas M. Munir

DOI: https://doi.org/10.1097/ico.0000000000003492

IF: 3.152

2024-02-23

Cornea

Abstract:Purpose: The aim of this study was to assess the capabilities of ChatGPT-4.0 and ChatGPT-3.5 for diagnosing corneal eye diseases based on case reports and compare with human experts. Methods: We randomly selected 20 cases of corneal diseases including corneal infections, dystrophies, and degenerations from a publicly accessible online database from the University of Iowa. We then input the text of each case description into ChatGPT-4.0 and ChatGPT-3.5 and asked for a provisional diagnosis. We finally evaluated the responses based on the correct diagnoses, compared them with the diagnoses made by 3 corneal specialists (human experts), and evaluated interobserver agreements. Results: The provisional diagnosis accuracy based on ChatGPT-4.0 was 85% (17 correct of 20 cases), whereas the accuracy of ChatGPT-3.5 was 60% (12 correct cases of 20). The accuracy of 3 corneal specialists compared with ChatGPT-4.0 and ChatGPT-3.5 was 100% (20 cases, P = 0.23, P = 0.0033), 90% (18 cases, P = 0.99, P = 0.6), and 90% (18 cases, P = 0.99, P = 0.6), respectively. The interobserver agreement between ChatGPT-4.0 and ChatGPT-3.5 was 65% (13 cases), whereas the interobserver agreement between ChatGPT-4.0 and 3 corneal specialists was 85% (17 cases), 80% (16 cases), and 75% (15 cases), respectively. However, the interobserver agreement between ChatGPT-3.5 and each of 3 corneal specialists was 60% (12 cases). Conclusions: The accuracy of ChatGPT-4.0 in diagnosing patients with various corneal conditions was markedly improved than ChatGPT-3.5 and promising for potential clinical integration. A balanced approach that combines artificial intelligence–generated insights with clinical expertise holds a key role for unveiling its full potential in eye care.

ophthalmology
Uncovering Language Disparity of ChatGPT on Retinal Vascular Disease Classification: Cross-Sectional Study

Xiaocong Liu,Jiageng Wu,An Shao,Wenyue Shen,Panpan Ye,Yao Wang,Juan Ye,Kai Jin,Jie Yang

DOI: https://doi.org/10.2196/51926

2024-01-22

Abstract:Background: Benefiting from rich knowledge and the exceptional ability to understand text, large language models like ChatGPT have shown great potential in English clinical environments. However, the performance of ChatGPT in non-English clinical settings, as well as its reasoning, have not been explored in depth. Objective: This study aimed to evaluate ChatGPT's diagnostic performance and inference abilities for retinal vascular diseases in a non-English clinical environment. Methods: In this cross-sectional study, we collected 1226 fundus fluorescein angiography reports and corresponding diagnoses written in Chinese and tested ChatGPT with 4 prompting strategies (direct diagnosis or diagnosis with a step-by-step reasoning process and in Chinese or English). Results: Compared with ChatGPT using Chinese prompts for direct diagnosis that achieved an F1-score of 70.47%, ChatGPT using English prompts for direct diagnosis achieved the best diagnostic performance (80.05%), which was inferior to ophthalmologists (89.35%) but close to ophthalmologist interns (82.69%). As for its inference abilities, although ChatGPT can derive a reasoning process with a low error rate (0.4 per report) for both Chinese and English prompts, ophthalmologists identified that the latter brought more reasoning steps with less incompleteness (44.31%), misinformation (1.96%), and hallucinations (0.59%) (all P<.001). Also, analysis of the robustness of ChatGPT with different language prompts indicated significant differences in the recall (P=.03) and F1-score (P=.04) between Chinese and English prompts. In short, when prompted in English, ChatGPT exhibited enhanced diagnostic and inference capabilities for retinal vascular disease classification based on Chinese fundus fluorescein angiography reports. Conclusions: ChatGPT can serve as a helpful medical assistant to provide diagnosis in non-English clinical environments, but there are still performance gaps, language disparities, and errors compared to professionals, which demonstrate the potential limitations and the need to continually explore more robust large language models in ophthalmology practice.

Bystander effects and radiotherapy.

Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images

Artificial intelligence chatbot interpretation of ophthalmic multimodal imaging cases

APPLICATIONS OF MULTIMODAL GENERATIVE ARTIFICIAL INTELLIGENCE IN A REAL-WORLD RETINA CLINIC SETTING

Uncovering Language Disparity of ChatGPT in Healthcare: Non-English Clinical Environment for Retinal Vascular Disease Classification

Performance Assessment of an Artificial Intelligence Chatbot in Clinical Vitreoretinal Scenarios

Diagnosing Glaucoma Based on the Ocular Hypertension Treatment Study Dataset Using Chat Generative Pre-Trained Transformer as a Large Language Model

Uncovering Language Disparity of ChatGPT in Healthcare: Non-English Clinical Environment for Retinal Vascular Disease Classification (Preprint)

ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

A User-friendly Approach for the Diagnosis of Diabetic Retinopathy Using ChatGPT and Automated Machine Learning

Applications of ChatGPT in the diagnosis, management, education, and research of retinal diseases: a scoping review

ChatGPT Assisting Diagnosis of Neuro-ophthalmology Diseases Based on Case Reports

Correlations between parameters of the social network and treatment outcomes of people suffering from schizophrenia seven years after the first hospitalization *

Evaluating the Artificial Intelligence Performance Growth in Ophthalmic Knowledge

Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis

Development and Evaluation of Aeyeconsult: A Novel Ophthalmology Chatbot Leveraging Verified Textbook Knowledge and GPT-4

Precision of artificial intelligence in paediatric cardiology multimodal image interpretation

Conversational AI Models for ophthalmic diagnosis: Comparison of ChatGPT and the Isabel Pro Differential Diagnosis Generator

Performance of ChatGPT in Diagnosis of Corneal Eye Diseases

Uncovering Language Disparity of ChatGPT on Retinal Vascular Disease Classification: Cross-Sectional Study