Abstract:Background: The availability of increasingly powerful large language models (LLMs) has attracted substantial interest in their potential for interpreting and generating human-like text for biomedical and clinical applications. However, there are often demands for high accuracy, concerns about balancing generalizability and domain-specificity, and questions about prompting robustness when considering the adoption of LLMs for specific use cases. There also is a lack of a framework or method to help choose which LLMs (or prompting strategies) should be adopted for specific biomedical or clinical tasks. Objective: To address the speculations on applying LLMs for biomedical applications, this study aims to 1) propose a framework to comprehensively evaluate and compare the performance of a range of LLMs and prompting techniques on a suite of biomedical natural language processing (NLP) tasks; 2) use the framework to benchmark several general- purpose LLMs and biomedical domain-specific LLMs. Methods: We evaluated and compared six general-purpose LLMs (GPT-4, GPT-3.5-Turbo, Flan-T5-XXL, Llama-3-8B-Instruct, Yi-1.5-34B-Chat, and Zephyr-7B-Beta) and three healthcare-specific LLMs (Medicine-Llama3-8B, Meditron-7B, and MedLLaMA-13B) on a set of 13 datasets - referred to as the Biomedical Language Understanding and Reasoning Benchmark (BLURB) - covering six commonly needed medical natural language processing tasks: named entity recognition (NER); relation extraction (RE); population, interventions, comparators, and outcomes (PICO); sentence similarity (SS); document classification (Class.); and question-answering (QA). All models were evaluated without further training or fine-tuning. Model performance was assessed according to a range of prompting strategies (formalized as a systematic, reusable prompting framework) and relied on the standard, task-specific evaluation metrics defined by BLURB. Results: Across all tasks, GPT-4 outperformed other LLMs, achieving a score of 64.6 on the benchmark, though other models, such as Flan-T5-XXL and Llama-3-8B-Instruct, demonstrated competitive performance on multiple tasks. We found that general-purpose models achieved better overall scores than domain-specific models, sometimes by significant margins. We observed a substantial impact of strategically editing the prompt describing the task and a consistent improvement in performance when including examples semantically similar to the input text. Additionally, the most performant prompts for nearly half the models outperformed the previously reported best results for the PubMedQA dataset from the BLURB leaderboard. Conclusions: These results provide evidence of the potential LLMs may have for biomedical applications and highlight the importance of robust evaluation before adopting LLMs for any specific use cases. Notably, performant open-source LLMs such as Llama-3- 8B-Instruct and Flan-T5-XXL show promise for use cases where trustworthiness and data confidentiality are concerns, as these models can be hosted locally, offering better security, transparency, and explainability. Continuing to explore how these emerging technologies can be adapted for the healthcare setting, paired with human expertise, and enhanced through quality control measures will be important research to allow responsible innovation with LLMs in the biomedical area.

Comparison of large language models for citation screening: A protocol for a prospective study

Performance of a Large Language Model in Screening Citations

Accuracy and reliability of data extraction for systematic reviews using large language models: A protocol for a prospective study

High-performance automated abstract screening with large language model ensembles

Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study

Large Language Model Augmented Clinical Trial Screening

Large language models for abstract screening in systematic- and scoping reviews: A diagnostic test accuracy study

A comparison of the diagnostic ability of large language models in challenging clinical cases

A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations

Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages

Leveraging large language models for systematic reviewing: A case study using HIV medication adherence research

The accuracy of large language models in labelling neurosurgical ‘case-control studies’ and risk of bias assessment: protocol for a study of interrater agreement with human reviewers.

Cutting Through the Clutter: The Potential of LLMs for Efficient Filtration in Systematic Literature Reviews

A Hybrid Semi-Automated Workflow for Systematic and Literature Review Processes with Large Language Model Analysis

Evaluation of Large Language Model Performance on the Biomedical Language Understanding and Reasoning Benchmark : Comparative Study

Practical Aspects of Using Large Language Models to Screen Abstracts for Cardiovascular Drug Development: Cross-Sectional Study

Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews

Benchmarking Large Language Models for Extraction of International Classification of Diseases Codes from Clinical Documentation

Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes

Zero-shot Generative Large Language Models for Systematic Review Screening Automation

Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks