Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words

Haochun Wang,Chi Liu,Nuwa Xi,Sendong Zhao,Meizhi Ju,Shiwei Zhang,Ziheng Zhang,Yefeng Zheng,Bing Qin,Ting Liu

2022-09-14

Abstract:Prompt-based fine-tuning for pre-trained models has proven effective for many natural language processing tasks under few-shot settings in general domain. However, tuning with prompt in biomedical domain has not been investigated thoroughly. Biomedical words are often rare in general domain, but quite ubiquitous in biomedical contexts, which dramatically deteriorates the performance of pre-trained models on downstream biomedical applications even after fine-tuning, especially in low-resource scenarios. We propose a simple yet effective approach to helping models learn rare biomedical words during tuning with prompt. Experimental results show that our method can achieve up to 6% improvement in biomedical natural language inference task without any extra parameters or training steps using few-shot vanilla prompt settings.

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the issue of pre-trained models struggling to understand rare biomedical terms in biomedical natural language processing tasks with a small number of labeled samples. Specifically, due to the limited amount of data in the biomedical field and the high cost of annotation, the performance of pre-trained models significantly declines when encountering rare biomedical terms. To solve this problem, the authors propose a simple and effective method that enhances the model's understanding of these terms by introducing definitions of rare terms during the fine-tuning process. Experimental results show that this method can significantly improve the model's performance in biomedical natural language inference tasks with a small number of samples, increasing performance by up to 6%, without requiring additional parameters or training steps.