Abstract:With the rapid progress in Natural Language Processing (NLP), Pre-trained Language Models (PLM) such as BERT, BioBERT, and ChatGPT have shown great potential in various medical NLP tasks. This paper surveys the cutting-edge achievements in applying PLMs to various medical NLP tasks. Specifically, we first brief PLMS and outline the research of PLMs in medicine. Next, we categorise and discuss the types of tasks in medical NLP, covering text summarisation, question-answering, machine translation, sentiment analysis, named entity recognition, information extraction, medical education, relation extraction, and text mining. For each type of task, we first provide an overview of the basic concepts, the main methodologies, the advantages of applying PLMs, the basic steps of applying PLMs application, the datasets for training and testing, and the metrics for task evaluation. Subsequently, a summary of recent important research findings is presented, analysing their motivations, strengths vs weaknesses, similarities vs differences, and discussing potential limitations. Also, we assess the quality and influence of the research reviewed in this paper by comparing the citation count of the papers reviewed and the reputation and impact of the conferences and journals where they are published. Through these indicators, we further identify the most concerned research topics currently. Finally, we look forward to future research directions, including enhancing models' reliability, explainability, and fairness, to promote the application of PLMs in clinical practice. In addition, this survey also collect some download links of some model codes and the relevant datasets, which are valuable references for researchers applying NLP techniques in medicine and medical professionals seeking to enhance their expertise and healthcare service through AI technology.

A Study of Pre-trained Language Models in Natural Language Processing

Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

Pre-Trained Language Models and Their Applications

Pre-trained language models in medicine: A survey

A Survey on Knowledge-Enhanced Pre-trained Language Models

Pre-trained models for natural language processing: A survey

ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Pre-train, Prompt and Recommendation: A Comprehensive Survey of Language Modelling Paradigm Adaptations in Recommender Systems

An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering

LERT: A Linguistically-motivated Pre-trained Language Model

On the Effectiveness of Pre-Trained Language Models for Legal Natural Language Processing: An Empirical Study

A Survey of Knowledge Enhanced Pre-trained Language Models

Pretrained Language Models for Text Generation: A Survey

Impossible Triangle: What's Next for Pre-trained Language Models?

Pre-trained Language Models for Text Generation: A Survey

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost