FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarios

Yongjian Tang,Rakebul Hasan,Thomas Runkler

2024-07-11

Abstract:Large Language Models (LLMs) have provided a new pathway for Named Entity Recognition (NER) tasks. Compared with fine-tuning, LLM-powered prompting methods avoid the need for training, conserve substantial computational resources, and rely on minimal annotated data. Previous studies have achieved comparable performance to fully supervised BERT-based fine-tuning approaches on general NER benchmarks. However, none of the previous approaches has investigated the efficiency of LLM-based few-shot learning in domain-specific scenarios. To address this gap, we introduce FsPONER, a novel approach for optimizing few-shot prompts, and evaluate its performance on domain-specific NER datasets, with a focus on industrial manufacturing and maintenance, while using multiple LLMs -- GPT-4-32K, GPT-3.5-Turbo, LLaMA 2-chat, and Vicuna. FsPONER consists of three few-shot selection methods based on random sampling, TF-IDF vectors, and a combination of both. We compare these methods with a general-purpose GPT-NER method as the number of few-shot examples increases and evaluate their optimal NER performance against fine-tuned BERT and LLaMA 2-chat. In the considered real-world scenarios with data scarcity, FsPONER with TF-IDF surpasses fine-tuned models by approximately 10% in F1 score.

Computation and Language,Information Retrieval

What problem does this paper attempt to address?

The paper aims to address the problem of optimizing prompts for Named Entity Recognition (NER) using a small amount of labeled data in specific domain scenarios. Specifically, the paper proposes FsPONER (Few-shot Prompt Optimization for Named Entity Recognition), a novel method for optimizing prompts with few examples, particularly for NER tasks in the industrial manufacturing and maintenance domains. By using several different few-shot example selection methods (random sampling, TF-IDF vectors, and a combination of both), FsPONER is able to improve NER performance in data-scarce situations. Compared to a fully supervised BERT model, FsPONER improves the F1 score by approximately 10% on domain-specific datasets when using the TF-IDF method. Additionally, the paper explores the impact of the number and scale of few-shot examples on NER performance and conducts experimental validation on multiple large language models (such as GPT-4-32K, GPT-3.5-Turbo, LLaMA 2-chat, and Vicuna).