The Analysis of the Difference between Infrared Soil Temperature and L Band Effective Soil Temperature

Hongzhang Ma,Qinhuo Liu

DOI: https://doi.org/10.1109/M2RSM.2011.5697425

2011-01-20

Abstract:NA

What problem does this paper attempt to address?

The Comparative Diagnostic Capability of Large Language Models in Otolaryngology

Akshay Warrier,Rohan Singh,Afash Haleem,Haider Zaki,Jean Anderson Eloy

DOI: https://doi.org/10.1002/lary.31434

IF: 2.97

2024-04-03

The Laryngoscope

Abstract:Objectives Evaluate and compare the ability of large language models (LLMs) to diagnose various ailments in otolaryngology. Methods We collected all 100 clinical vignettes from the second edition of Otolaryngology Cases—The University of Cincinnati Clinical Portfolio by Pensak et al. With the addition of the prompt "Provide a diagnosis given the following history," we prompted ChatGPT‐3.5, Google Bard, and Bing‐GPT4 to provide a diagnosis for each vignette. These diagnoses were compared to the portfolio for accuracy and recorded. All queries were run in June 2023. Results ChatGPT‐3.5 was the most accurate model (89% success rate), followed by Google Bard (82%) and Bing GPT (74%). A chi‐squared test revealed a significant difference between the three LLMs in providing correct diagnoses (p = 0.023). Of the 100 vignettes, seven require additional testing results (i.e., biopsy, non‐contrast CT) for accurate clinical diagnosis. When omitting these vignettes, the revised success rates were 95.7% for ChatGPT‐3.5, 88.17% for Google Bard, and 78.72% for Bing‐GPT4 (p = 0.002). Conclusions ChatGPT‐3.5 offers the most accurate diagnoses when given established clinical vignettes as compared to Google Bard and Bing‐GPT4. LLMs may accurately offer assessments for common otolaryngology conditions but currently require detailed prompt information and critical supervision from clinicians. There is vast potential in the clinical applicability of LLMs; however, practitioners should be wary of possible "hallucinations" and misinformation in responses. Level of Evidence 3 Laryngoscope, 2024

medicine, research & experimental,otorhinolaryngology
Comparative Evaluation of LLMs in Clinical Oncology

Nicholas R Rydzewski,Deepak Dinakaran,Shuang G Zhao,Eytan Ruppin,Baris Turkbey,Deborah E Citrin,Krishnan R Patel

DOI: https://doi.org/10.1056/aioa2300151

Abstract:Background: As artificial intelligence (AI) tools become widely accessible, more patients and medical professionals will turn to them for medical information. Large language models (LLMs), a subset of AI, excel in natural language processing tasks and hold considerable promise for clinical use. Fields such as oncology, in which clinical decisions are highly dependent on a continuous influx of new clinical trial data and evolving guidelines, stand to gain immensely from such advancements. It is therefore of critical importance to benchmark these models and describe their performance characteristics to guide their safe application to clinical oncology. Accordingly, the primary objectives of this work were to conduct comprehensive evaluations of LLMs in the field of oncology and to identify and characterize strategies that medical professionals can use to bolster their confidence in a model's response. Methods: This study tested five publicly available LLMs (LLaMA 1, PaLM 2, Claude-v1, generative pretrained transformer 3.5 [GPT-3.5], and GPT-4) on a comprehensive battery of 2044 oncology questions, including topics from medical oncology, surgical oncology, radiation oncology, medical statistics, medical physics, and cancer biology. Model prompts were presented independently of each other, and each prompt was repeated three times to assess output consistency. For each response, models were instructed to provide a self-appraised confidence score (from 1 to 4). Model performance was also evaluated against a novel validation set comprising 50 oncology questions curated to eliminate any risk of overlap with the data used to train the LLMs. Results: There was significant heterogeneity in performance between models (analysis of variance, P<0.001). Relative to a human benchmark (2013 and 2014 examination results), GPT-4 was the only model to perform above the 50th percentile. Overall, model performance varied as a function of subject area across all models, with worse performance observed in clinical oncology subcategories compared with foundational topics (medical statistics, medical physics, and cancer biology). Within the clinical oncology subdomain, worse performance was observed in female-predominant malignancies. A combination of model selection, prompt repetition, and confidence self-appraisal allowed for the identification of high-performing subgroups of questions with observed accuracies of 81.7 and 81.1% in the Claude-v1 and GPT-4 models, respectively. Evaluation of the novel validation question set produced similar trends in model performance while also highlighting improved performance in newer, centrally hosted models (GPT-4 Turbo and Gemini 1.0 Ultra) and local models (Mixtral 8×7B and LLaMA 2). Conclusions: Of the models tested on a standardized set of oncology questions, GPT-4 was observed to have the highest performance. Although this performance is impressive, all LLMs continue to have clinically significant error rates, including examples of overconfidence and consistent inaccuracies. Given the enthusiasm to integrate these new implementations of AI into clinical practice, continued standardized evaluations of the strengths and limitations of these products will be critical to guide both patients and medical professionals. (Funded by the National Institutes of Health Clinical Center for Research and the Intramural Research Program of the National Institutes of Health; Z99 CA999999.).
Evaluation of large language models as a diagnostic aid for complex medical cases

Alejandro Ríos-Hoyo,Naing Lin Shan,Anran Li,Alexander T. Pearson,Lajos Pusztai,Frederick M. Howard

DOI: https://doi.org/10.3389/fmed.2024.1380148

IF: 3.9

2024-06-20

Frontiers in Medicine

Abstract:Background The use of large language models (LLM) has recently gained popularity in diverse areas, including answering questions posted by patients as well as medical professionals. Objective To evaluate the performance and limitations of LLMs in providing the correct diagnosis for a complex clinical case. Design Seventy-five consecutive clinical cases were selected from the Massachusetts General Hospital Case Records, and differential diagnoses were generated by OpenAI’s GPT3.5 and 4 models. Results The mean number of diagnoses provided by the Massachusetts General Hospital case discussants was 16.77, by GPT3.5 30 and by GPT4 15.45 ( p < 0.0001). GPT4 was more frequently able to list the correct diagnosis as first (22% versus 20% with GPT3.5, p = 0.86), provide the correct diagnosis among the top three generated diagnoses (42% versus 24%, p = 0.075). GPT4 was better at providing the correct diagnosis, when the different diagnoses were classified into groups according to the medical specialty and include the correct diagnosis at any point in the differential list (68% versus 48%, p = 0.0063). GPT4 provided a differential list that was more similar to the list provided by the case discussants than GPT3.5 (Jaccard Similarity Index 0.22 versus 0.12, p = 0.001). Inclusion of the correct diagnosis in the generated differential was correlated with PubMed articles matching the diagnosis (OR 1.40, 95% CI 1.25–1.56 for GPT3.5, OR 1.25, 95% CI 1.13–1.40 for GPT4), but not with disease incidence. Conclusions and relevance The GPT4 model was able to generate a differential diagnosis list with the correct diagnosis in approximately two thirds of cases, but the most likely diagnosis was often incorrect for both models. In its current state, this tool can at most be used as an aid to expand on potential diagnostic considerations for a case, and future LLMs should be trained which account for the discrepancy between disease incidence and availability in the literature.

medicine, general & internal
The Diagnostic Performance of Large Language Models and General Radiologists in Thoracic Radiology Cases: A Comparative Study

Yasin Celal Gunes,Turay Cesur

DOI: https://doi.org/10.1097/RTI.0000000000000805

2024-09-13

Abstract:Purpose: To investigate and compare the diagnostic performance of 10 different large language models (LLMs) and 2 board-certified general radiologists in thoracic radiology cases published by The Society of Thoracic Radiology. Materials and methods: We collected publicly available 124 "Case of the Month" from the Society of Thoracic Radiology website between March 2012 and December 2023. Medical history and imaging findings were input into LLMs for diagnosis and differential diagnosis, while radiologists independently visually provided their assessments. Cases were categorized anatomically (parenchyma, airways, mediastinum-pleura-chest wall, and vascular) and further classified as specific or nonspecific for radiologic diagnosis. Diagnostic accuracy and differential diagnosis scores (DDxScore) were analyzed using the χ2, Kruskal-Wallis, Wilcoxon, McNemar, and Mann-Whitney U tests. Results: Among the 124 cases, Claude 3 Opus showed the highest diagnostic accuracy (70.29%), followed by ChatGPT 4/Google Gemini 1.5 Pro (59.75%), Meta Llama 3 70b (57.3%), ChatGPT 3.5 (53.2%), outperforming radiologists (52.4% and 41.1%) and other LLMs (P<0.05). Claude 3 Opus DDxScore was significantly better than other LLMs and radiologists, except ChatGPT 3.5 (P<0.05). All LLMs and radiologists showed greater accuracy in specific cases (P<0.05), with no DDxScore difference for Perplexity and Google Bard based on specificity (P>0.05). There were no significant differences between LLMs and radiologists in the diagnostic accuracy of anatomic subgroups (P>0.05), except for Meta Llama 3 70b in the vascular cases (P=0.040). Conclusions: Claude 3 Opus outperformed other LLMs and radiologists in text-based thoracic radiology cases. LLMs hold great promise for clinical decision systems under proper medical supervision.
Evaluating large language models on medical, lay language, and self-reported descriptions of genetic conditions

Kendall A Flaharty,Ping Hu,Suzanna Ledgister Hanchard,Molly E Ripper,Dat Duong,Rebekah L Waikel,Benjamin D Solomon

DOI: https://doi.org/10.1016/j.ajhg.2024.07.011

2024-07-31

Abstract:Large language models (LLMs) are generating interest in medical settings. For example, LLMs can respond coherently to medical queries by providing plausible differential diagnoses based on clinical notes. However, there are many questions to explore, such as evaluating differences between open- and closed-source LLMs as well as LLM performance on queries from both medical and non-medical users. In this study, we assessed multiple LLMs, including Llama-2-chat, Vicuna, Medllama2, Bard/Gemini, Claude, ChatGPT3.5, and ChatGPT-4, as well as non-LLM approaches (Google search and Phenomizer) regarding their ability to identify genetic conditions from textbook-like clinician questions and their corresponding layperson translations related to 63 genetic conditions. For open-source LLMs, larger models were more accurate than smaller LLMs: 7b, 13b, and larger than 33b parameter models obtained accuracy ranges from 21%-49%, 41%-51%, and 54%-68%, respectively. Closed-source LLMs outperformed open-source LLMs, with ChatGPT-4 performing best (89%-90%). Three of 11 LLMs and Google search had significant performance gaps between clinician and layperson prompts. We also evaluated how in-context prompting and keyword removal affected open-source LLM performance. Models were provided with 2 types of in-context prompts: list-type prompts, which improved LLM performance, and definition-type prompts, which did not. We further analyzed removal of rare terms from descriptions, which decreased accuracy for 5 of 7 evaluated LLMs. Finally, we observed much lower performance with real individuals' descriptions; LLMs answered these questions with a maximum 21% accuracy.
Benchmarking the Confidence of Large Language Models in Clinical Questions

Mahmud Omar Sr.,Reem Agbareia,Benjamin S Glicksberg Sr.,Girish Nadkarni,Eyal Klang

DOI: https://doi.org/10.1101/2024.08.11.24311810

2024-09-10

Abstract:Background and Aim: Large language models (LLMs) show promise in healthcare, but their self-assessment capabilities remain unclear. This study evaluates the confidence levels and performance of 12 LLMs across five medical specialties to assess their ability to accurately judge their responses. Methods: We used 1965 multiple-choice questions from internal medicine, obstetrics and gynecology, psychiatry, pediatrics, and general surgery. Models were prompted to provide answers and confidence scores. Performance and confidence were analyzed using chi-square tests and t-tests. Consistency across question versions was also evaluated. Results: All models displayed high confidence regardless of answer correctness. Higher-tier models showed slightly better calibration, with a mean confidence of 72.5% for correct answers versus 69.4% for incorrect ones, compared to lower-tier models (79.6% vs 79.5%). The mean confidence difference between correct and incorrect responses ranged from 0.6% to 5.4% across all models. Four models showed significantly higher confidence when correct (p<0.01), but the difference remained small. Most models demonstrated consistency across question versions. Conclusion: While newer LLMs show improved performance and consistency in medical knowledge tasks, their confidence levels remain poorly calibrated. The gap between performance and self-assessment poses risks in clinical applications. Until these models can reliably gauge their certainty, their use in healthcare should be limited and supervised by experts. Further research on human-AI collaboration and ensemble methods is needed for responsible implementation. Keywords: Large Language Models (LLMs), Safe AI, AI Reliability, Clinical knowledge.
CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

Mingyu Derek Ma,Chenchen Ye,Yu Yan,Xiaoxuan Wang,Peipei Ping,Timothy S Chang,Wei Wang

2024-10-12

Abstract:The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophisticated, patient-specific decisions need to be made. Current evaluations of LLMs in this field are often narrow in scope, focusing on specific diseases or specialties and employing simplified diagnostic tasks. To bridge this gap, we introduce CliBench, a novel benchmark developed from the MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. This benchmark not only covers diagnoses from a diverse range of medical cases across various specialties but also incorporates tasks of clinical significance: treatment procedure identification, lab test ordering and medication prescriptions. Supported by structured output ontologies, CliBench enables a precise and multi-granular evaluation, offering an in-depth understanding of LLM's capability on diverse clinical tasks of desired granularity. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making. Our preliminary results shed light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.

Computation and Language,Artificial Intelligence,Machine Learning
Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes

Balu Bhasuran,Qiao Jin,Yuzhang Xie,Carl Yang,Karim Hanna,Jennifer Costa,Cindy Shavor,Zhiyong Lu,Zhe He

2024-11-01

Abstract:Differential diagnosis is crucial for medicine as it helps healthcare providers systematically distinguish between conditions that share similar symptoms. This study assesses the impact of lab test results on differential diagnoses (DDx) made by large language models (LLMs). Clinical vignettes from 50 case reports from PubMed Central were created incorporating patient demographics, symptoms, and lab results. Five LLMs GPT-4, GPT-3.5, Llama-2-70b, Claude-2, and Mixtral-8x7B were tested to generate Top 10, Top 5, and Top 1 DDx with and without lab data. A comprehensive evaluation involving GPT-4, a knowledge graph, and clinicians was conducted. GPT-4 performed best, achieving 55% accuracy for Top 1 diagnoses and 60% for Top 10 with lab data, with lenient accuracy up to 80%. Lab results significantly improved accuracy, with GPT-4 and Mixtral excelling, though exact match rates were low. Lab tests, including liver function, metabolic/toxicology panels, and serology/immune tests, were generally interpreted correctly by LLMs for differential diagnosis.

Computation and Language,Artificial Intelligence
On the limitations of large language models in clinical diagnosis

D. Danis,C. Mungall,J. H. Caufield,E. Casiraghi,J. Reese,Peter N. Robinson,T. Groza,G. Valentini

DOI: https://doi.org/10.1101/2023.07.13.23292613

2023-07-14

MedRxiv

Abstract:Background: The potential of large language models (LLM) such as GPT to support complex tasks such as differential diagnosis has been a subject of debate, with some ascribing near sentient abilities to the models and others claiming that LLMs merely perform "autocomplete on steroids". A recent study reported that the Generative Pretrained Transformer 4 (GPT-4) model performed well in complex differential diagnostic reasoning. The authors assessed the performance of GPT-4 in identifying the correct diagnosis in a series of case records from the New England Journal of Medicine. The authors constructed prompts based on the clinical presentation section of the case reports, and compared the results of GPT-4 to the actual diagnosis. GPT-4 returned the correct diagnosis as a part of its response in 64% of cases, with the correct diagnosis being at rank 1 in 39% of cases. However, such concise but comprehensive narratives of the clinical course are not typically available in electronic health records (EHRs). Further, if they were available, EHR records contain identifying information whose transmission is prohibited by Health Insurance Portability and Accountability Act (HIPAA) regulations. Methods: To assess the expected performance of GPT on comparable datasets that can be generated by text mining and by design cannot contain identifiable information, we parsed the texts of the case reports and extracted Human Phenotype Ontology (HPO) terms, from which prompts for GPT were constructed that contain largely the same clinical abnormalities but lack the surrounding narrative. Results: While the performance of GPT-4 on the original narrative-based text was good, with the final diagnosis being included in its differential in 29/75 cases (38.7%; rank 1 in 17.3% of cases; mean rank of 3.4), the performance of GPT-4 on the feature-based approach that includes the major clinical abnormalities without additional narrative texas substantially worse, with GPT-4 including the final diagnosis in its differential in 8/75 cases (10.7%; rank 1 in 4.0% of cases; mean rank of 3.9). Interpretation: We consider the feature-based queries to be a more appropriate test of the performance of GPT-4 in diagnostic tasks, since it is unlikely that the narrative approach can be used in actual clinical practice. Future research and algorithmic development is needed to determine the optimal approach to leveraging LLMs for clinical diagnosis.

Computer Science,Medicine
Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks

Sarah Sandmann,Sarah Riepenhausen,Lucas Plagwitz,Julian Varghese

DOI: https://doi.org/10.1038/s41467-024-46411-8

IF: 16.6

2024-03-06

Nature Communications

Abstract:Abstract It is likely that individuals are turning to Large Language Models (LLMs) to seek health advice, much like searching for diagnoses on Google. We evaluate clinical accuracy of GPT-3·5 and GPT-4 for suggesting initial diagnosis, examination steps and treatment of 110 medical cases across diverse clinical disciplines. Moreover, two model configurations of the Llama 2 open source LLMs are assessed in a sub-study. For benchmarking the diagnostic task, we conduct a naïve Google search for comparison. Overall, GPT-4 performed best with superior performances over GPT-3·5 considering diagnosis and examination and superior performance over Google for diagnosis. Except for treatment, better performance on frequent vs rare diseases is evident for all three approaches. The sub-study indicates slightly lower performances for Llama models. In conclusion, the commercial LLMs show growing potential for medical question answering in two successive major releases. However, some weaknesses underscore the need for robust and regulated AI models in health care. Open source LLMs can be a viable option to address specific needs regarding data privacy and transparency of training.

multidisciplinary sciences
Learning to Make Rare and Complex Diagnoses With Generative AI Assistance: Qualitative Study of Popular Large Language Models

Tassallah Abdullahi,Ritambhara Singh,Carsten Eickhoff

DOI: https://doi.org/10.2196/51391

2024-02-13

Abstract:Background: Patients with rare and complex diseases often experience delayed diagnoses and misdiagnoses because comprehensive knowledge about these diseases is limited to only a few medical experts. In this context, large language models (LLMs) have emerged as powerful knowledge aggregation tools with applications in clinical decision support and education domains. Objective: This study aims to explore the potential of 3 popular LLMs, namely Bard (Google LLC), ChatGPT-3.5 (OpenAI), and GPT-4 (OpenAI), in medical education to enhance the diagnosis of rare and complex diseases while investigating the impact of prompt engineering on their performance. Methods: We conducted experiments on publicly available complex and rare cases to achieve these objectives. We implemented various prompt strategies to evaluate the performance of these models using both open-ended and multiple-choice prompts. In addition, we used a majority voting strategy to leverage diverse reasoning paths within language models, aiming to enhance their reliability. Furthermore, we compared their performance with the performance of human respondents and MedAlpaca, a generative LLM specifically designed for medical tasks. Results: Notably, all LLMs outperformed the average human consensus and MedAlpaca, with a minimum margin of 5% and 13%, respectively, across all 30 cases from the diagnostic case challenge collection. On the frequently misdiagnosed cases category, Bard tied with MedAlpaca but surpassed the human average consensus by 14%, whereas GPT-4 and ChatGPT-3.5 outperformed MedAlpaca and the human respondents on the moderately often misdiagnosed cases category with minimum accuracy scores of 28% and 11%, respectively. The majority voting strategy, particularly with GPT-4, demonstrated the highest overall score across all cases from the diagnostic complex case collection, surpassing that of other LLMs. On the Medical Information Mart for Intensive Care-III data sets, Bard and GPT-4 achieved the highest diagnostic accuracy scores, with multiple-choice prompts scoring 93%, whereas ChatGPT-3.5 and MedAlpaca scored 73% and 47%, respectively. Furthermore, our results demonstrate that there is no one-size-fits-all prompting approach for improving the performance of LLMs and that a single strategy does not universally apply to all LLMs. Conclusions: Our findings shed light on the diagnostic capabilities of LLMs and the challenges associated with identifying an optimal prompting strategy that aligns with each language model's characteristics and specific task requirements. The significance of prompt engineering is highlighted, providing valuable insights for researchers and practitioners who use these language models for medical training. Furthermore, this study represents a crucial step toward understanding how LLMs can enhance diagnostic reasoning in rare and complex medical cases, paving the way for developing effective educational tools and accurate diagnostic aids to improve patient care and outcomes.
Evaluating the Diagnostic Performance of Large Language Models in Identifying Complex Multisystemic Syndromes: A Comparative Study with Radiology Residents

Hagar Olshaker,Dana Brin,Elinor Kalderon,Matan Kraus,Eliahu Konen,Eyal Klang

DOI: https://doi.org/10.1101/2024.06.05.24308335

2024-06-07

Abstract:Aim: This study evaluates the diagnostic capabilities of large language models (LLMs) in interpreting imaging patterns, focusing on their utility as a resource for radiology residents. We compare the diagnostic performance of OpenAI's GPT-3.5, GPT-4, and Google's Gemini Pro against radiology residents in identifying complex, multisystemic syndromes with an increased risk of cancer. Methods: We assessed diagnostic accuracy using textual descriptions of radiological findings from 60 diseases selected from The Familial Cancer Database. Participants included three LLMs and three radiology residents. Diagnostic responses were scored on accuracy and first choice correctness. Experiments with AI models were conducted using default API settings. Results: GPT-4 achieved the highest diagnostic accuracy (63%) and first choice accuracy (40%), significantly outperforming the radiology residents whose accuracy ranged from 22% to 43%. The overall average accuracy for AI models was 49.3%, compared to 29.0% for residents. Error analysis revealed that while some diseases were universally recognized, others highlighted diagnostic challenges across both human and AI participants. Conclusion: GPT-4 outperforms radiology residents in diagnosing complex, infrequent multisystemic diseases. These findings suggest potential benefits of integrating AI tools to improve diagnostic accuracy for rare conditions and imply a need for revisions in medical training to incorporate AI competencies, enhancing diagnostic processes and resident education in radiology.
Digital Diagnostics: The Potential Of Large Language Models In Recognizing Symptoms Of Common Illnesses

Gaurav Kumar Gupta,Aditi Singh,Sijo Valayakkad Manikandan,Abul Ehtesham

DOI: https://doi.org/10.48550/arxiv.2405.06712

2024-05-09

Computation and Language

Abstract:The recent swift development of LLMs like GPT-4, Gemini, and GPT-3.5 offers a transformative opportunity in medicine and healthcare, especially in digital diagnostics. This study evaluates each model diagnostic abilities by interpreting a user symptoms and determining diagnoses that fit well with common illnesses, and it demonstrates how each of these models could significantly increase diagnostic accuracy and efficiency. Through a series of diagnostic prompts based on symptoms from medical databases, GPT-4 demonstrates higher diagnostic accuracy from its deep and complete history of training on medical data. Meanwhile, Gemini performs with high precision as a critical tool in disease triage, demonstrating its potential to be a reliable model when physicians are trying to make high-risk diagnoses. GPT-3.5, though slightly less advanced, is a good tool for medical diagnostics. This study highlights the need to study LLMs for healthcare and clinical practices with more care and attention, ensuring that any system utilizing LLMs promotes patient privacy and complies with health information privacy laws such as HIPAA compliance, as well as the social consequences that affect the varied individuals in complex healthcare contexts. This study marks the start of a larger future effort to study the various ways in which assigning ethical concerns to LLMs task of learning from human biases could unearth new ways to apply AI in complex medical settings.
Towards Accurate Differential Diagnosis with Large Language Models

Daniel McDuff,Mike Schaekermann,Tao Tu,Anil Palepu,Amy Wang,Jake Garrison,Karan Singhal,Yash Sharma,Shekoofeh Azizi,Kavita Kulkarni,Le Hou,Yong Cheng,Yun Liu,S Sara Mahdavi,Sushant Prakash,Anupam Pathak,Christopher Semturs,Shwetak Patel,Dale R Webster,Ewa Dominowska,Juraj Gottweis,Joelle Barral,Katherine Chou,Greg S Corrado,Yossi Matias,Jake Sunshine,Alan Karthikesalingam,Vivek Natarajan

2023-12-01

Abstract:An accurate differential diagnosis (DDx) is a cornerstone of medical care, often reached through an iterative process of interpretation that combines clinical history, physical examination, investigations and procedures. Interactive interfaces powered by Large Language Models (LLMs) present new opportunities to both assist and automate aspects of this process. In this study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its ability to generate a DDx alone or as an aid to clinicians. 20 clinicians evaluated 302 challenging, real-world medical cases sourced from the New England Journal of Medicine (NEJM) case reports. Each case report was read by two clinicians, who were randomized to one of two assistive conditions: either assistance from search engines and standard medical resources, or LLM assistance in addition to these tools. All clinicians provided a baseline, unassisted DDx prior to using the respective assistive tools. Our LLM for DDx exhibited standalone performance that exceeded that of unassisted clinicians (top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study arms, the DDx quality score was higher for clinicians assisted by our LLM (top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%) (McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p = 0.03). Further, clinicians assisted by our LLM arrived at more comprehensive differential lists than those without its assistance. Our study suggests that our LLM for DDx has potential to improve clinicians' diagnostic reasoning and accuracy in challenging cases, meriting further real-world evaluation for its ability to empower physicians and widen patients' access to specialist-level expertise.

Computers and Society,Artificial Intelligence
Large language models in solving clinical dilemmas - advantages and drawbacks

Obla Amar Bapu,D.,Juan,J.,Gahleitner,F.,Macleod,K.,Urquhart,D. S.,Mcdougall,C.,Unger,S. A.,Armstrong,D.,Narayanan,M.,Juan,J.

DOI: https://doi.org/10.1183/13993003.congress-2024.pa4379

IF: 24.3

2024-11-01

European Respiratory Journal

Abstract:Background: Large language models (LLMs) have shown potential to assist clinical decisions but concerns about underlying mechanism ('black box phenomenon') have provoked unease. Aims: To explore the LLM black box in the context of complex decision making in paediatric respiratory medicine (PRM) by comparison against trainee doctors (TDs), qualitative assessment of responses and a deep-dive into their sources. Methods: Six complex PRM scenarios were posed to 10 TDs and 3 LLMs. Six PRM experts provided detailed comments on the responses, which were analysed qualitatively using DisplayRTM. Screen recordings of TDs' search and sources provided by LLMs were analysed for further insight. Results: Word-cloud analysis shows features of LLM and TD responses (Table). ChatGPTTM provided useful responses but did not quote sources. TDs sourced responses from patient-facing websites, abstracts and review articles (Figure), while BardTM and BingTM quoted journals or evidence-based reviews. ChatGPTBardBingTDsStructureGoodGoodAdequatePoorOmissionsNoOccasionalYesVariableNew AdvancesPre-2021YesYesVariableIncorrectNoNoNoNoSentiment score0.85–0.15–1.0–0.34 Conclusion: LLMs (particularly ChatGPT and Bard) already surpass TDs in quality of responses. We demonstrate utility of LLMs for non-expert clinicians faced with complex medical scenarios. We explore the drawbacks of the individual LLMs while noting that this is a rapidly changing area.

respiratory system
Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Paul Hager,Friederike Jungmann,Kunal Bhagat,Inga Hubrecht,Manuel Knauer,Jakob Vielhauer,Robbie Holland,Rickmer Braren,Marcus Makowski,Georgios Kaisis,Daniel Rueckert

DOI: https://doi.org/10.1101/2024.01.26.24301810

2024-01-26

Abstract:Clinical decision making is one of the most impactful parts of a physician’s responsibilities and stands to benefit greatly from AI solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills that are necessary for deployment in a realistic clinical decision making environment, including gathering information, adhering to established guidelines, and integrating into clinical workflows. To understand how useful LLMs are in real-world settings, we must evaluate them , i.e. on real-world data under realistic conditions. Here we have created a curated dataset based on the MIMIC-IV database spanning 2400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians on average), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for clinical deployment while providing a dataset and framework to guide future studies.

Health Informatics
Evaluation of large language models in breast cancer clinical scenarios: A comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

Linfang Deng,Tianyi Wang,Yangzhang,Zhenhua Zhai,Wei Tao,Jincheng Li,Yi Zhao,Jinjiang Xu,Shaoting Luo

DOI: https://doi.org/10.1097/js9.0000000000001066

2024-01-23

International Journal of Surgery

Abstract:Background Large language models (LLMs) have garnered significant attention in the AI domain owing to their exemplary context recognition and response capabilities. However, the potential of LLMs in specific clinical scenarios, particularly in breast cancer diagnosis, treatment, and care, has not been fully explored. This study aimed to compare the performances of three major LLMs in the clinical context of breast cancer. Methods In this study, clinical scenarios designed specifically for breast cancer were segmented into five pivotal domains (nine cases): assessment and diagnosis, treatment decision-making, post-operative care, psychosocial support, and prognosis and rehabilitation. The LLMs were used to generate feedback for various queries related to these domains. For each scenario, a panel of five breast cancer specialists, each with over a decade of experience, evaluated the feedback from LLMs. They assessed feedback concerning LLMs in terms of their quality, relevance, and applicability. Results There was a moderate level of agreement among the raters ( Fleiss’ kappa =0.345, P <0.05). Comparing the performance of different models regarding response length, GPT-4.0 and GPT-3.5 provided relatively longer feedback than Claude2. Furthermore, across the nine case analyses, GPT-4.0 significantly outperformed the other two models in average quality, relevance, and applicability. Within the five clinical areas, GPT-4.0 markedly surpassed GPT-3.5 in the quality of the other four areas and scored higher than Claude2 in tasks related to psychosocial support and treatment decision-making. Conclusion This study revealed that in the realm of clinical applications for breast cancer, GPT-4.0 showcases not only superiority in terms of quality and relevance but also demonstrates exceptional capability in applicability, especially when compared to GPT-3.5. Relative to Claude2, GPT-4.0 holds advantages in specific domains. With the expanding use of LLMs in the clinical field, ongoing optimization and rigorous accuracy assessments are paramount.

surgery
Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

Ankit Pal,Malaikannan Sankarasubbu

2024-02-11

Abstract:Large language models have the potential to be valuable in the healthcare industry, but it's crucial to verify their safety and effectiveness through rigorous evaluation. For this purpose, we comprehensively evaluated both open-source LLMs and Google's new multimodal LLM called Gemini across Medical reasoning, hallucination detection, and Medical Visual Question Answering tasks. While Gemini showed competence, it lagged behind state-of-the-art models like MedPaLM 2 and GPT-4 in diagnostic accuracy. Additionally, Gemini achieved an accuracy of 61.45\% on the medical VQA dataset, significantly lower than GPT-4V's score of 88\%. Our analysis revealed that Gemini is highly susceptible to hallucinations, overconfidence, and knowledge gaps, which indicate risks if deployed uncritically. We also performed a detailed analysis by medical subject and test type, providing actionable feedback for developers and clinicians. To mitigate risks, we applied prompting strategies that improved performance. Additionally, we facilitated future research and development by releasing a Python module for medical LLM evaluation and establishing a dedicated leaderboard on Hugging Face for medical domain LLMs. Python module can be found at

Artificial Intelligence,Computer Vision and Pattern Recognition,Human-Computer Interaction,Machine Learning
A Survey of Clinicians’ Views of the Utility of Large Language Models

Matthew Spotnitz,Betina Idnay,Emily R Gordon,Rebecca Shyu,Gongbo Zhang,Cong Liu,James J Cimino,Chunhua Weng

DOI: https://doi.org/10.1055/a-2281-7092

IF: 2.762

2024-03-05

Applied Clinical Informatics

Abstract:Objective: Large language models (LLMs) like ChatGPT are powerful algorithms that have been shown to produce human-like text from input data. A number of potential clinical applications of this technology have been proposed and evaluated by biomedical informatics experts. However, few have surveyed healthcare providers for their opinions about whether the technology is fit for use. Materials and Methods: We distributed a validated mixed-methods survey to gauge practicing clinicians’ comfort with LLMs for a breadth of tasks in clinical practice, research and education, which were selected from the literature. Results: A total of 30 clinicians fully completed the survey. Of the 23 tasks, 16 were rated positively by more than 50% of the respondents. Based on our qualitative analysis, healthcare providers considered LLMs to have excellent synthesis skills and efficiency. However, our respondents had concerns that LLMs could generate false information and propagate training data bias. Discussion: Our survey respondents were most comfortable with scenarios that allow LLMs to function in an assistive role, like a physician extender or trainee. Conclusion: In a uniquely rigorous and comprehensive mixed-methods survey of clinicians about LLM use, healthcare providers were encouraging of having LLMs in healthcare for many tasks, and especially in assistive roles. There is a need for continued human-centered development of both LLMs and artificial intelligence (AI) in general.

medical informatics
A Comprehensive Evaluation of Large Language Models on Mental Illnesses

Abdelrahman Hanafi,Mohammed Saad,Noureldin Zahran,Radwa J. Hanafy,Mohammed E. Fouda

2024-09-24

Abstract:Large language models have shown promise in various domains, including healthcare. In this study, we conduct a comprehensive evaluation of LLMs in the context of mental health tasks using social media data. We explore the zero-shot (ZS) and few-shot (FS) capabilities of various LLMs, including GPT-4, Llama 3, Gemini, and others, on tasks such as binary disorder detection, disorder severity evaluation, and psychiatric knowledge assessment. Our evaluation involved 33 models testing 9 main prompt templates across the tasks. Key findings revealed that models like GPT-4 and Llama 3 exhibited superior performance in binary disorder detection, with accuracies reaching up to 85% on certain datasets. Moreover, prompt engineering played a crucial role in enhancing model performance. Notably, the Mixtral 8x22b model showed an improvement of over 20%, while Gemma 7b experienced a similar boost in performance. In the task of disorder severity evaluation, we observed that FS learning significantly improved the model's accuracy, highlighting the importance of contextual examples in complex assessments. Notably, the Phi-3-mini model exhibited a substantial increase in performance, with balanced accuracy improving by over 6.80% and mean average error dropping by nearly 1.3 when moving from ZS to FS learning. In the psychiatric knowledge task, recent models generally outperformed older, larger counterparts, with the Llama 3.1 405b achieving an accuracy of 91.2%. Despite promising results, our analysis identified several challenges, including variability in performance across datasets and the need for careful prompt engineering. Furthermore, the ethical guards imposed by many LLM providers hamper the ability to accurately evaluate their performance, due to tendency to not respond to potentially sensitive queries.

Artificial Intelligence

The Analysis of the Difference between Infrared Soil Temperature and L Band Effective Soil Temperature

The Comparative Diagnostic Capability of Large Language Models in Otolaryngology

Comparative Evaluation of LLMs in Clinical Oncology

Evaluation of large language models as a diagnostic aid for complex medical cases

The Diagnostic Performance of Large Language Models and General Radiologists in Thoracic Radiology Cases: A Comparative Study

Evaluating large language models on medical, lay language, and self-reported descriptions of genetic conditions

Benchmarking the Confidence of Large Language Models in Clinical Questions

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes

On the limitations of large language models in clinical diagnosis

Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks

Learning to Make Rare and Complex Diagnoses With Generative AI Assistance: Qualitative Study of Popular Large Language Models

Evaluating the Diagnostic Performance of Large Language Models in Identifying Complex Multisystemic Syndromes: A Comparative Study with Radiology Residents

Digital Diagnostics: The Potential Of Large Language Models In Recognizing Symptoms Of Common Illnesses

Towards Accurate Differential Diagnosis with Large Language Models

Large language models in solving clinical dilemmas - advantages and drawbacks

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Evaluation of large language models in breast cancer clinical scenarios: A comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations

A Survey of Clinicians’ Views of the Utility of Large Language Models

A Comprehensive Evaluation of Large Language Models on Mental Illnesses