Abstract:Background: Large language models (LLMs) have shown remarkable capabilities in natural language processing (NLP), especially in domains where labeled data are scarce or expensive, such as the clinical domain. However, to unlock the clinical knowledge hidden in these LLMs, we need to design effective prompts that can guide them to perform specific clinical NLP tasks without any task-specific training data. This is known as in-context learning, which is an art and science that requires understanding the strengths and weaknesses of different LLMs and prompt engineering approaches. Objective: The objective of this study is to assess the effectiveness of various prompt engineering techniques, including 2 newly introduced types—heuristic and ensemble prompts, for zero-shot and few-shot clinical information extraction using pretrained language models. Methods: This comprehensive experimental study evaluated different prompt types (simple prefix, simple cloze, chain of thought, anticipatory, heuristic, and ensemble) across 5 clinical NLP tasks: clinical sense disambiguation, biomedical evidence extraction, coreference resolution, medication status extraction, and medication attribute extraction. The performance of these prompts was assessed using 3 state-of-the-art language models: GPT-3.5 (OpenAI), Gemini (Google), and LLaMA-2 (Meta). The study contrasted zero-shot with few-shot prompting and explored the effectiveness of ensemble approaches. Results: The study revealed that task-specific prompt tailoring is vital for the high performance of LLMs for zero-shot clinical NLP. In clinical sense disambiguation, GPT-3.5 achieved an accuracy of 0.96 with heuristic prompts and 0.94 in biomedical evidence extraction. Heuristic prompts, alongside chain of thought prompts, were highly effective across tasks. Few-shot prompting improved performance in complex scenarios, and ensemble approaches capitalized on multiple prompt strengths. GPT-3.5 consistently outperformed Gemini and LLaMA-2 across tasks and prompt types. Conclusions: This study provides a rigorous evaluation of prompt engineering methodologies and introduces innovative techniques for clinical information extraction, demonstrating the potential of in-context learning in the clinical domain. These findings offer clear guidelines for future prompt-based clinical NLP research, facilitating engagement by non-NLP experts in clinical NLP advancements. To the best of our knowledge, this is one of the first works on the empirical evaluation of different prompt engineering approaches for clinical NLP in this era of generative artificial intelligence, and we hope that it will inspire and inform future research in this area.

Zero-Shot Information Extraction for Clinical Meta-Analysis using Large Language Models

Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models

Large Language Models are Few-Shot Clinical Information Extractors

Large Language Models for Efficient Medical Information Extraction

Zero-shot Generative Large Language Models for Systematic Review Screening Automation

Almanac: Retrieval-Augmented Language Models for Clinical Medicine

Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

High-performance automated abstract screening with large language model ensembles

Transformative potential of Large Language Models in data mining on Electronic Health Records.

Zero-Shot Clinical Trial Patient Matching with LLMs

Using large language models for safety-related table summarization in clinical study reports

Zero-Shot LLMs for Named Entity Recognition: Targeting Cardiac Function Indicators in German Clinical Texts

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology

Large Language Models Streamline Automated Machine Learning for Clinical Studies

Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study

Leveraging Large Language Models for Medical Information Extraction and Query Generation

Enhancing Clinical Data Extraction from Pathology Reports: A Comparative Analysis of Large Language Models

An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study

A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks