Abstract:Background: Large language models (LLMs) have shown remarkable capabilities in natural language processing (NLP), especially in domains where labeled data are scarce or expensive, such as the clinical domain. However, to unlock the clinical knowledge hidden in these LLMs, we need to design effective prompts that can guide them to perform specific clinical NLP tasks without any task-specific training data. This is known as in-context learning, which is an art and science that requires understanding the strengths and weaknesses of different LLMs and prompt engineering approaches. Objective: The objective of this study is to assess the effectiveness of various prompt engineering techniques, including 2 newly introduced types—heuristic and ensemble prompts, for zero-shot and few-shot clinical information extraction using pretrained language models. Methods: This comprehensive experimental study evaluated different prompt types (simple prefix, simple cloze, chain of thought, anticipatory, heuristic, and ensemble) across 5 clinical NLP tasks: clinical sense disambiguation, biomedical evidence extraction, coreference resolution, medication status extraction, and medication attribute extraction. The performance of these prompts was assessed using 3 state-of-the-art language models: GPT-3.5 (OpenAI), Gemini (Google), and LLaMA-2 (Meta). The study contrasted zero-shot with few-shot prompting and explored the effectiveness of ensemble approaches. Results: The study revealed that task-specific prompt tailoring is vital for the high performance of LLMs for zero-shot clinical NLP. In clinical sense disambiguation, GPT-3.5 achieved an accuracy of 0.96 with heuristic prompts and 0.94 in biomedical evidence extraction. Heuristic prompts, alongside chain of thought prompts, were highly effective across tasks. Few-shot prompting improved performance in complex scenarios, and ensemble approaches capitalized on multiple prompt strengths. GPT-3.5 consistently outperformed Gemini and LLaMA-2 across tasks and prompt types. Conclusions: This study provides a rigorous evaluation of prompt engineering methodologies and introduces innovative techniques for clinical information extraction, demonstrating the potential of in-context learning in the clinical domain. These findings offer clear guidelines for future prompt-based clinical NLP research, facilitating engagement by non-NLP experts in clinical NLP advancements. To the best of our knowledge, this is one of the first works on the empirical evaluation of different prompt engineering approaches for clinical NLP in this era of generative artificial intelligence, and we hope that it will inspire and inform future research in this area.

Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting

A Benchmark Evaluation of Clinical Named Entity Recognition in French

Large Language Models Struggle in Token-Level Clinical Named Entity Recognition

LLMs in Biomedicine: A study on clinical Named Entity Recognition

An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study

A comparative analysis of Spanish Clinical encoder-based models on NER and classification tasks

How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain

Clinical information extraction for lower-resource languages and domains with few-shot learning using pretrained language models and prompting

Language inference-based learning for Low-Resource Chinese clinical named entity recognition using language model

Ensemble of Deep Masked Language Models for Effective Named Entity Recognition in Health and Life Science Corpora

SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials

Beyond Tokens: Fair Evaluation of French Large Language Models for Clinical Named Entity Recognition

PromptNER: Prompting For Named Entity Recognition

Improving Large Language Models for Clinical Named Entity Recognition via Prompt Engineering

Clinical Prompt Learning With Frozen Language Models

Large Language Models Vote: Prompting for Rare Disease Identification

MANER: Mask Augmented Named Entity Recognition for Extreme Low-Resource Languages

Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting

Evaluating Named Entity Recognition Using Few-Shot Prompting with Large Language Models