Abstract:Background: Large language models (LLMs) have the potential to support promising new applications in health informatics. However, practical data on sample size considerations for fine-tuning LLMs to perform specific tasks in biomedical and health policy contexts are lacking. Objective: This study aims to evaluate sample size and sample selection techniques for fine-tuning LLMs to support improved named entity recognition (NER) for a custom data set of conflicts of interest disclosure statements. Methods: A random sample of 200 disclosure statements was prepared for annotation. All "PERSON" and "ORG" entities were identified by each of the 2 raters, and once appropriate agreement was established, the annotators independently annotated an additional 290 disclosure statements. From the 490 annotated documents, 2500 stratified random samples in different size ranges were drawn. The 2500 training set subsamples were used to fine-tune a selection of language models across 2 model architectures (Bidirectional Encoder Representations from Transformers [BERT] and Generative Pre-trained Transformer [GPT]) for improved NER, and multiple regression was used to assess the relationship between sample size (sentences), entity density (entities per sentence [EPS]), and trained model performance (F1-score). Additionally, single-predictor threshold regression models were used to evaluate the possibility of diminishing marginal returns from increased sample size or entity density. Results: Fine-tuned models ranged in topline NER performance from F1-score=0.79 to F1-score=0.96 across architectures. Two-predictor multiple linear regression models were statistically significant with multiple R2 ranging from 0.6057 to 0.7896 (all P<.001). EPS and the number of sentences were significant predictors of F1-scores in all cases ( P<.001), except for the GPT-2_large model, where EPS was not a significant predictor (P=.184). Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large. Likewise, the threshold regression models indicate a diminishing marginal return for EPS with point estimates between 1.36 and 1.38. Conclusions: Relatively modest sample sizes can be used to fine-tune LLMs for NER tasks applied to biomedical text, and training data entity density should representatively approximate entity density in production data. Training data quality and a model architecture's intended use (text generation vs text processing or classification) may be as, or more, important as training data volume and model parameter size.

LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty

LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty

Incorporating Large Language Models into Named Entity Recognition: Opportunities and Challenges

LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data

GPT-NER: Named Entity Recognition via Large Language Models

Advancing entity recognition in biomedicine via instruction tuning of large language models

Large Language Models Struggle in Token-Level Clinical Named Entity Recognition

Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognition

GEIC: Universal and Multilingual Named Entity Recognition with Large Language Models

Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning

E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition

Augmenting NER Datasets with LLMs: Towards Automated and Refined Annotation

Joint Entity Recognition and Disambiguation.

A Knowledge-Enhanced Medical Named Entity Recognition Method that Integrates Pre-Trained Language Models

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies

CLLMFS: A Contrastive Learning enhanced Large Language Model Framework for Few-Shot Named Entity Recognition

Sample Size Considerations for Fine-Tuning Large Language Models for Named Entity Recognition Tasks: Methodological Study

On Leveraging Large Language Models for Enhancing Entity Resolution: A Cost-efficient Approach

VANER: Leveraging Large Language Model for Versatile and Adaptive Biomedical Named Entity Recognition