Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions

Tianyu Han,Lisa C. Adams,Keno K. Bressem,Felix Busch,Sven Nebelung,Daniel Truhn

DOI: https://doi.org/10.1001/jama.2023.27861

IF: 11.816

2024-03-18

JAMA

Abstract:This study compares 2 large language models and their performance vs that of competing open-source models.

medicine, general & internal

What problem does this paper attempt to address?

Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases

Wan Hang Keith Chiu,Wei Sum Koel Ko,William Chi Shing Cho,Sin Yu Joanne Hui,Wing Chi Lawrence Chan,Michael D Kuo

DOI: https://doi.org/10.2196/53724

IF: 7.076

2024-05-13

Journal of Medical Internet Research

Abstract:Large language models showed interpretative reasoning in solving diagnostically challenging medical cases.

health care sciences & services,medical informatics
Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Fenglin Liu,Zheng Li,Hongjian Zhou,Qingyu Yin,Jingfeng Yang,Xianfeng Tang,Chen Luo,Ming Zeng,Haoming Jiang,Yifan Gao,Priyanka Nigam,Sreyashi Nag,Bing Yin,Yining Hua,Xuan Zhou,Omid Rohanian,Anshul Thakur,Lei Clifton,David Clifton

DOI: https://doi.org/10.1101/2024.04.24.24306315

2024-10-16

Abstract:The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering (QA) task with answer options for evaluation. However, many clinical decisions involve answering open-ended questions without pre-set options. To better understand LLMs in the clinic, we construct a benchmark ClinicBench. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. Furthermore, we construct six novel datasets and clinical tasks that are complex but common in real-world practice, e.g., open-ended decision-making, long document processing, and emerging drug analysis. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings. Finally, we invite medical experts to evaluate the clinical usefulness of LLMs. The benchmark data is available at https://github.com/AI-in-Health/ClinicBench.
A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Jiaqi Wang,Hanqi Jiang,Yiheng Liu,Chong Ma,Xu Zhang,Yi Pan,Mengyuan Liu,Peiran Gu,Sichen Xia,Wenjun Li,Yutong Zhang,Zihao Wu,Zhengliang Liu,Tianyang Zhong,Bao Ge,Tuo Zhang,Ning Qiang,Xintao Hu,Xi Jiang,Xin Zhang,Wei Zhang,Dinggang Shen,Tianming Liu,Shu Zhang

2024-08-02

Abstract:In an era defined by the explosive growth of data and rapid technological advancements, Multimodal Large Language Models (MLLMs) stand at the forefront of artificial intelligence (AI) systems. Designed to seamlessly integrate diverse data types-including text, images, videos, audio, and physiological sequences-MLLMs address the complexities of real-world applications far beyond the capabilities of single-modality systems. In this paper, we systematically sort out the applications of MLLM in multimodal tasks such as natural language, vision, and audio. We also provide a comparative analysis of the focus of different MLLMs in the tasks, and provide insights into the shortcomings of current MLLMs, and suggest potential directions for future research. Through these discussions, this paper hopes to provide valuable insights for the further development and application of MLLM.

Artificial Intelligence
Large Language Models in Healthcare: A Comprehensive Benchmark

Andrew Liu,Hongjian Zhou,Yining Hua,Omid Rohanian,Lei Clifton,David A. Clifton

2024-04-25

Abstract:The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering task with answer options for evaluation. However, in real clinical settings, many clinical decisions, such as treatment recommendations, involve answering open-ended questions without pre-set options. Meanwhile, existing studies mainly use accuracy to assess model performance. In this paper, we comprehensively benchmark diverse LLMs in healthcare, to clearly understand their strengths and weaknesses. Our benchmark contains seven tasks and thirteen datasets across medical language generation, understanding, and reasoning. We conduct a detailed evaluation of the existing sixteen LLMs in healthcare under both zero-shot and few-shot (i.e., 1,3,5-shot) learning settings. We report the results on five metrics (i.e. matching, faithfulness, comprehensiveness, generalizability, and robustness) that are critical in achieving trust from clinical users. We further invite medical experts to conduct human evaluation.

Computation and Language,Artificial Intelligence
Multiple Choice Questions and Large Languages Models: A Case Study with Fictional Medical Data

Maxime Griot,Jean Vanderdonckt,Demet Yuksel,Coralie Hemptinne

2024-06-04

Abstract:Large Language Models (LLMs) like ChatGPT demonstrate significant potential in the medical field, often evaluated using multiple-choice questions (MCQs) similar to those found on the USMLE. Despite their prevalence in medical education, MCQs have limitations that might be exacerbated when assessing LLMs. To evaluate the effectiveness of MCQs in assessing the performance of LLMs, we developed a fictional medical benchmark focused on a non-existent gland, the Glianorex. This approach allowed us to isolate the knowledge of the LLM from its test-taking abilities. We used GPT-4 to generate a comprehensive textbook on the Glianorex in both English and French and developed corresponding multiple-choice questions in both languages. We evaluated various open-source, proprietary, and domain-specific LLMs using these questions in a zero-shot setting. The models achieved average scores around 67%, with minor performance differences between larger and smaller models. Performance was slightly higher in English than in French. Fine-tuned medical models showed some improvement over their base versions in English but not in French. The uniformly high performance across models suggests that traditional MCQ-based benchmarks may not accurately measure LLMs' clinical knowledge and reasoning abilities, instead highlighting their pattern recognition skills. This study underscores the need for more robust evaluation methods to better assess the true capabilities of LLMs in medical contexts.

Computation and Language,Artificial Intelligence,Machine Learning
ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation

Guangyu Wang,Guoxing Yang,Zongxin Du,Longjun Fan,Xiaohu Li

2023-06-17

Abstract:Large language models have exhibited exceptional performance on various Natural Language Processing (NLP) tasks, leveraging techniques such as the pre-training, and instruction fine-tuning. Despite these advances, their effectiveness in medical applications is limited, due to challenges such as factual inaccuracies, reasoning abilities, and lack grounding in real-world experience. In this study, we present ClinicalGPT, a language model explicitly designed and optimized for clinical scenarios. By incorporating extensive and diverse real-world data, such as medical records, domain-specific knowledge, and multi-round dialogue consultations in the training process, ClinicalGPT is better prepared to handle multiple clinical task. Furthermore, we introduce a comprehensive evaluation framework that includes medical knowledge question-answering, medical exams, patient consultations, and diagnostic analysis of medical records. Our results demonstrate that ClinicalGPT significantly outperforms other models in these tasks, highlighting the effectiveness of our approach in adapting large language models to the critical domain of healthcare.

Computation and Language
Performance of Large Language Models on Medical Oncology Examination Questions

Jack B. Longwell,Ian Hirsch,Fernando Binder,Galileo Arturo Gonzalez Conchas,Daniel Mau,Raymond Jang,Rahul G. Krishnan,Robert C. Grant

DOI: https://doi.org/10.1001/jamanetworkopen.2024.17641

2024-06-19

JAMA Network Open

Abstract:This cross-sectional study evaluates the accuracy of large language model (LLM) answers to examination-style multiple choice medical oncology questions and assessed whether errors in LLM responses would be likely to cause harm.

medicine, general & internal
Evaluation of large language models in breast cancer clinical scenarios: A comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

Linfang Deng,Tianyi Wang,Yangzhang,Zhenhua Zhai,Wei Tao,Jincheng Li,Yi Zhao,Jinjiang Xu,Shaoting Luo

DOI: https://doi.org/10.1097/js9.0000000000001066

2024-01-23

International Journal of Surgery

Abstract:Background Large language models (LLMs) have garnered significant attention in the AI domain owing to their exemplary context recognition and response capabilities. However, the potential of LLMs in specific clinical scenarios, particularly in breast cancer diagnosis, treatment, and care, has not been fully explored. This study aimed to compare the performances of three major LLMs in the clinical context of breast cancer. Methods In this study, clinical scenarios designed specifically for breast cancer were segmented into five pivotal domains (nine cases): assessment and diagnosis, treatment decision-making, post-operative care, psychosocial support, and prognosis and rehabilitation. The LLMs were used to generate feedback for various queries related to these domains. For each scenario, a panel of five breast cancer specialists, each with over a decade of experience, evaluated the feedback from LLMs. They assessed feedback concerning LLMs in terms of their quality, relevance, and applicability. Results There was a moderate level of agreement among the raters ( Fleiss’ kappa =0.345, P <0.05). Comparing the performance of different models regarding response length, GPT-4.0 and GPT-3.5 provided relatively longer feedback than Claude2. Furthermore, across the nine case analyses, GPT-4.0 significantly outperformed the other two models in average quality, relevance, and applicability. Within the five clinical areas, GPT-4.0 markedly surpassed GPT-3.5 in the quality of the other four areas and scored higher than Claude2 in tasks related to psychosocial support and treatment decision-making. Conclusion This study revealed that in the realm of clinical applications for breast cancer, GPT-4.0 showcases not only superiority in terms of quality and relevance but also demonstrates exceptional capability in applicability, especially when compared to GPT-3.5. Relative to Claude2, GPT-4.0 holds advantages in specific domains. With the expanding use of LLMs in the clinical field, ongoing optimization and rigorous accuracy assessments are paramount.

surgery
The Analysis of the Difference between Infrared Soil Temperature and L Band Effective Soil Temperature

Hongzhang Ma,Qinhuo Liu

DOI: https://doi.org/10.1109/M2RSM.2011.5697425

2011-01-20

Abstract:NA
The Comparative Diagnostic Capability of Large Language Models in Otolaryngology

Akshay Warrier,Rohan Singh,Afash Haleem,Haider Zaki,Jean Anderson Eloy

DOI: https://doi.org/10.1002/lary.31434

IF: 2.97

2024-04-03

The Laryngoscope

Abstract:Objectives Evaluate and compare the ability of large language models (LLMs) to diagnose various ailments in otolaryngology. Methods We collected all 100 clinical vignettes from the second edition of Otolaryngology Cases—The University of Cincinnati Clinical Portfolio by Pensak et al. With the addition of the prompt "Provide a diagnosis given the following history," we prompted ChatGPT‐3.5, Google Bard, and Bing‐GPT4 to provide a diagnosis for each vignette. These diagnoses were compared to the portfolio for accuracy and recorded. All queries were run in June 2023. Results ChatGPT‐3.5 was the most accurate model (89% success rate), followed by Google Bard (82%) and Bing GPT (74%). A chi‐squared test revealed a significant difference between the three LLMs in providing correct diagnoses (p = 0.023). Of the 100 vignettes, seven require additional testing results (i.e., biopsy, non‐contrast CT) for accurate clinical diagnosis. When omitting these vignettes, the revised success rates were 95.7% for ChatGPT‐3.5, 88.17% for Google Bard, and 78.72% for Bing‐GPT4 (p = 0.002). Conclusions ChatGPT‐3.5 offers the most accurate diagnoses when given established clinical vignettes as compared to Google Bard and Bing‐GPT4. LLMs may accurately offer assessments for common otolaryngology conditions but currently require detailed prompt information and critical supervision from clinicians. There is vast potential in the clinical applicability of LLMs; however, practitioners should be wary of possible "hallucinations" and misinformation in responses. Level of Evidence 3 Laryngoscope, 2024

medicine, research & experimental,otorhinolaryngology
Large language models encode clinical knowledge

Karan Singhal,Shekoofeh Azizi,Tao Tu,S. Sara Mahdavi,Jason Wei,Hyung Won Chung,Nathan Scales,Ajay Tanwani,Heather Cole-Lewis,Stephen Pfohl,Perry Payne,Martin Seneviratne,Paul Gamble,Chris Kelly,Abubakr Babiker,Nathanael Schärli,Aakanksha Chowdhery,Philip Mansfield,Dina Demner-Fushman,Blaise Agüera y Arcas,Dale Webster,Greg S. Corrado,Yossi Matias,Katherine Chou,Juraj Gottweis,Nenad Tomasev,Yun Liu,Alvin Rajkomar,Joelle Barral,Christopher Semturs,Alan Karthikesalingam,Vivek Natarajan

DOI: https://doi.org/10.1038/s41586-023-06291-2

IF: 64.8

2023-07-13

Nature

Abstract:Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

multidisciplinary sciences
Large Language Model as a Universal Clinical Multi-task Decoder

Yujiang Wu,Hongjian Song,Jiawen Zhang,Xumeng Wen,Shun Zheng,Jiang Bian

2024-06-18

Abstract:The development of effective machine learning methodologies for enhancing the efficiency and accuracy of clinical systems is crucial. Despite significant research efforts, managing a plethora of diversified clinical tasks and adapting to emerging new tasks remain significant challenges. This paper presents a novel paradigm that employs a pre-trained large language model as a universal clinical multi-task decoder. This approach leverages the flexibility and diversity of language expressions to handle task topic variations and associated arguments. The introduction of a new task simply requires the addition of a new instruction template. We validate this framework across hundreds of tasks, demonstrating its robustness in facilitating multi-task predictions, performing on par with traditional multi-task learning and single-task learning approaches. Moreover, it shows exceptional adaptability to new tasks, with impressive zero-shot performance in some instances and superior data efficiency in few-shot scenarios. This novel approach offers a unified solution to manage a wide array of new and emerging tasks in clinical applications.

Computation and Language,Artificial Intelligence
CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

Zetian Ouyang,Yishuai Qiu,Linlin Wang,Gerard de Melo,Ya Zhang,Yanfeng Wang,Liang He

2024-10-04

Abstract:With the proliferation of Large Language Models (LLMs) in diverse domains, there is a particular need for unified evaluation standards in clinical medical scenarios, where models need to be examined very thoroughly. We present CliMedBench, a comprehensive benchmark with 14 expert-guided core clinical scenarios specifically designed to assess the medical ability of LLMs across 7 pivot dimensions. It comprises 33,735 questions derived from real-world medical reports of top-tier tertiary hospitals and authentic examination exercises. The reliability of this benchmark has been confirmed in several ways. Subsequent experiments with existing LLMs have led to the following findings: (i) Chinese medical LLMs underperform on this benchmark, especially where medical reasoning and factual consistency are vital, underscoring the need for advances in clinical knowledge and diagnostic accuracy. (ii) Several general-domain LLMs demonstrate substantial potential in medical clinics, while the limited input capacity of many medical LLMs hinders their practical use. These findings reveal both the strengths and limitations of LLMs in clinical scenarios and offer critical insights for medical research.

Computation and Language
Evaluating large language models on medical, lay language, and self-reported descriptions of genetic conditions

Kendall A Flaharty,Ping Hu,Suzanna Ledgister Hanchard,Molly E Ripper,Dat Duong,Rebekah L Waikel,Benjamin D Solomon

DOI: https://doi.org/10.1016/j.ajhg.2024.07.011

2024-07-31

Abstract:Large language models (LLMs) are generating interest in medical settings. For example, LLMs can respond coherently to medical queries by providing plausible differential diagnoses based on clinical notes. However, there are many questions to explore, such as evaluating differences between open- and closed-source LLMs as well as LLM performance on queries from both medical and non-medical users. In this study, we assessed multiple LLMs, including Llama-2-chat, Vicuna, Medllama2, Bard/Gemini, Claude, ChatGPT3.5, and ChatGPT-4, as well as non-LLM approaches (Google search and Phenomizer) regarding their ability to identify genetic conditions from textbook-like clinician questions and their corresponding layperson translations related to 63 genetic conditions. For open-source LLMs, larger models were more accurate than smaller LLMs: 7b, 13b, and larger than 33b parameter models obtained accuracy ranges from 21%-49%, 41%-51%, and 54%-68%, respectively. Closed-source LLMs outperformed open-source LLMs, with ChatGPT-4 performing best (89%-90%). Three of 11 LLMs and Google search had significant performance gaps between clinician and layperson prompts. We also evaluated how in-context prompting and keyword removal affected open-source LLM performance. Models were provided with 2 types of in-context prompts: list-type prompts, which improved LLM performance, and definition-type prompts, which did not. We further analyzed removal of rare terms from descriptions, which decreased accuracy for 5 of 7 evaluated LLMs. Finally, we observed much lower performance with real individuals' descriptions; LLMs answered these questions with a maximum 21% accuracy.
Multimodal Large Language Models: A Survey

Philip S. Yu,Shicheng Wan,Zefeng Chen,Jiayang Wu,Wensheng Gan

DOI: https://doi.org/10.1109/BigData59044.2023.10386743

2023-11-22

Abstract:The exploration of multimodal language models integrates multiple data types, such as images, text, language, audio, and other heterogeneity. While the latest large language models excel in text-based tasks, they often struggle to understand and process other data types. Multimodal models address this limitation by combining various modalities, enabling a more comprehensive understanding of diverse data. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. Furthermore, we introduce a range of multimodal products, focusing on the efforts of major technology companies. A practical guide is provided, offering insights into the technical aspects of multimodal models. Moreover, we present a compilation of the latest algorithms and commonly used datasets, providing researchers with valuable resources for experimentation and evaluation. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development. By addressing these aspects, this paper aims to facilitate a deeper understanding of multimodal models and their potentiality in various domains.

Computer Science,Linguistics
A Survey on Benchmarks of Multimodal Large Language Models

Jian Li,Weiheng Lu,Hao Fei,Meng Luo,Ming Dai,Min Xia,Yizhang Jin,Zhenye Gan,Ding Qi,Chaoyou Fu,Ying Tai,Wankou Yang,Yabiao Wang,Chengjie Wang

2024-09-06

Abstract:Multimodal Large Language Models (MLLMs) are gaining increasing popularity in both academia and industry due to their remarkable performance in various applications such as visual question answering, visual perception, understanding, and reasoning. Over the past few years, significant efforts have been made to examine MLLMs from multiple perspectives. This paper presents a comprehensive review of 200 benchmarks and evaluations for MLLMs, focusing on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Finally, we discuss the limitations of the current evaluation methods for MLLMs and explore promising future directions. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better. For more details, please visit our GitHub repository: <a class="link-external link-https" href="https://github.com/swordlidev/Evaluation-Multimodal-LLMs-Survey" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition
Cross-Modal Consistency in Multimodal Large Language Models

Xiang Zhang,Senyu Li,Ning Shi,Bradley Hauer,Zijun Wu,Grzegorz Kondrak,Muhammad Abdul-Mageed,Laks V.S. Lakshmanan

2024-11-14

Abstract:Recent developments in multimodal methodologies have marked the beginning of an exciting era for models adept at processing diverse data types, encompassing text, audio, and visual content. Models like GPT-4V, which merge computer vision with advanced language processing, exhibit extraordinary proficiency in handling intricate tasks that require a simultaneous understanding of both textual and visual information. Prior research efforts have meticulously evaluated the efficacy of these Vision Large Language Models (VLLMs) in various domains, including object detection, image captioning, and other related fields. However, existing analyses have often suffered from limitations, primarily centering on the isolated evaluation of each modality's performance while neglecting to explore their intricate cross-modal interactions. Specifically, the question of whether these models achieve the same level of accuracy when confronted with identical task instances across different modalities remains unanswered. In this study, we take the initiative to delve into the interaction and comparison among these modalities of interest by introducing a novel concept termed cross-modal consistency. Furthermore, we propose a quantitative evaluation framework founded on this concept. Our experimental findings, drawn from a curated collection of parallel vision-language datasets developed by us, unveil a pronounced inconsistency between the vision and language modalities within GPT-4V, despite its portrayal as a unified multimodal model. Our research yields insights into the appropriate utilization of such models and hints at potential avenues for enhancing their design.

Computation and Language,Artificial Intelligence
Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in Clinical Notes

Xinsong Du,John Novoa-Laurentiev,Joseph M Plasaek,Ya-Wen Chuang,Liqin Wang,Gad Marshall,Stephanie K Mueller,Frank Chang,Surabhi Datta,Hunki Paek,Bin Lin,Qiang Wei,Xiaoyan Wang,Jingqi Wang,Hao Ding,Frank J Manion,Jingcheng Du,David W Bates,Li Zhou

DOI: https://doi.org/10.1101/2024.04.03.24305298

2024-05-06

MedRxiv

Abstract:Background: Large language models (LLMs) have shown promising performance in various healthcare domains, but their effectiveness in identifying specific clinical conditions in real medical records is less explored. This study evaluates LLMs for detecting signs of cognitive decline in real electronic health record (EHR) clinical notes, comparing their error profiles with traditional models. The insights gained will inform strategies for performance enhancement. Methods: This study, conducted at Mass General Brigham in Boston, MA, analyzed clinical notes from the four years prior to a 2019 diagnosis of mild cognitive impairment in patients aged 50 and older. We used a randomly annotated sample of 4,949 note sections, filtered with keywords related to cognitive functions, for model development. For testing, a random annotated sample of 1,996 note sections without keyword filtering was utilized. We developed prompts for two LLMs, Llama 2 and GPT-4, on HIPAA-compliant cloud-computing platforms using multiple approaches (e.g., both hard and soft prompting and error analysis-based instructions) to select the optimal LLM-based method. Baseline models included a hierarchical attention-based neural network and XGBoost. Subsequently, we constructed an ensemble of the three models using a majority vote approach. Results: GPT-4 demonstrated superior accuracy and efficiency compared to Llama 2, but did not outperform traditional models. The ensemble model outperformed the individual models, achieving a precision of 90.3%, a recall of 94.2%, and an F1-score of 92.2%. Notably, the ensemble model showed a significant improvement in precision, increasing from a range of 70%-79% to above 90%, compared to the best-performing single model. Error analysis revealed that 63 samples were incorrectly predicted by at least one model; however, only 2 cases (3.2%) were mutual errors across all models, indicating diverse error profiles among them. Conclusions: LLMs and traditional machine learning models trained using local EHR data exhibited diverse error profiles. The ensemble of these models was found to be complementary, enhancing diagnostic performance. Future research should investigate integrating LLMs with smaller, localized models and incorporating medical data and domain knowledge to enhance performance on specific tasks.
Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes

Darren Liu,Cheng Ding,Delgersuren Bold,Monique Bouvier,Jiaying Lu,Benjamin Shickel,Craig S. Jabaley,Wenhui Zhang,Soojin Park,Michael J. Young,Mark S. Wainwright,Gilles Clermont,Parisa Rashidi,Eric S. Rosenthal,Laurie Dimisko,Ran Xiao,Joo Heung Yoon,Carl Yang,Xiao Hu

2024-01-25

Abstract:The field of healthcare has increasingly turned its focus towards Large Language Models (LLMs) due to their remarkable performance. However, their performance in actual clinical applications has been underexplored. Traditional evaluations based on question-answering tasks don't fully capture the nuanced contexts. This gap highlights the need for more in-depth and practical assessments of LLMs in real-world healthcare settings. Objective: We sought to evaluate the performance of LLMs in the complex clinical context of adult critical care medicine using systematic and comprehensible analytic methods, including clinician annotation and adjudication. Methods: We investigated the performance of three general LLMs in understanding and processing real-world clinical notes. Concepts from 150 clinical notes were identified by MetaMap and then labeled by 9 clinicians. Each LLM's proficiency was evaluated by identifying the temporality and negation of these concepts using different prompts for an in-depth analysis. Results: GPT-4 showed overall superior performance compared to other LLMs. In contrast, both GPT-3.5 and text-davinci-003 exhibit enhanced performance when the appropriate prompting strategies are employed. The GPT family models have demonstrated considerable efficiency, evidenced by their cost-effectiveness and time-saving capabilities. Conclusion: A comprehensive qualitative performance evaluation framework for LLMs is developed and operationalized. This framework goes beyond singular performance aspects. With expert annotations, this methodology not only validates LLMs' capabilities in processing complex medical data but also establishes a benchmark for future LLM evaluations across specialized domains.

Computation and Language,Artificial Intelligence,Software Engineering
CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

Mingyu Derek Ma,Chenchen Ye,Yu Yan,Xiaoxuan Wang,Peipei Ping,Timothy S Chang,Wei Wang

2024-10-12

Abstract:The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophisticated, patient-specific decisions need to be made. Current evaluations of LLMs in this field are often narrow in scope, focusing on specific diseases or specialties and employing simplified diagnostic tasks. To bridge this gap, we introduce CliBench, a novel benchmark developed from the MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. This benchmark not only covers diagnoses from a diverse range of medical cases across various specialties but also incorporates tasks of clinical significance: treatment procedure identification, lab test ordering and medication prescriptions. Supported by structured output ontologies, CliBench enables a precise and multi-granular evaluation, offering an in-depth understanding of LLM's capability on diverse clinical tasks of desired granularity. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making. Our preliminary results shed light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.

Computation and Language,Artificial Intelligence,Machine Learning

Comparative Analysis of Multimodal Large Language Model Performance on Clinical Vignette Questions

Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Large Language Models in Healthcare: A Comprehensive Benchmark

Multiple Choice Questions and Large Languages Models: A Case Study with Fictional Medical Data

ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation

Performance of Large Language Models on Medical Oncology Examination Questions

Evaluation of large language models in breast cancer clinical scenarios: A comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2

The Analysis of the Difference between Infrared Soil Temperature and L Band Effective Soil Temperature

The Comparative Diagnostic Capability of Large Language Models in Otolaryngology

Large language models encode clinical knowledge

Large Language Model as a Universal Clinical Multi-task Decoder

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

Evaluating large language models on medical, lay language, and self-reported descriptions of genetic conditions

Multimodal Large Language Models: A Survey

A Survey on Benchmarks of Multimodal Large Language Models

Cross-Modal Consistency in Multimodal Large Language Models

Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in Clinical Notes

Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making