Abstract:Specialised pre-trained language models are becoming more frequent in Natural language Processing (NLP) since they can potentially outperform models trained on generic texts. BioBERT (Sanh et al., Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01108, 2019) and BioClinicalBERT (Alsentzer et al., Publicly available clinical bert embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72–78, 2019) are two examples of such models that have shown promise in medical NLP tasks. Many of these models are overparametrised and resource-intensive, but thanks to techniques like knowledge distillation, it is possible to create smaller versions that perform almost as well as their larger counterparts. In this work, we specifically focus on development of compact language models for processing clinical texts (i.e. progress notes, discharge summaries, etc). We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning, with the number of parameters ranging from million to million. These models performed comparably to larger models such as BioBERT and ClinicalBioBERT and significantly outperformed other compact models trained on general or biomedical data. Our extensive evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks, including natural language inference, relation extraction, named entity recognition and sequence classification. To our knowledge, this is the first comprehensive study specifically focused on creating efficient and compact transformers for clinical NLP tasks. The models and code used in this study can be found on our Huggingface profile at https://huggingface.co/nlpie and Github page at https://github.com/nlpie-research/Lightweight-Clinical-Transformers, respectively, promoting reproducibility of our results.

Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Tasks

Efficiency at scale: Investigating the performance of diminutive language models in clinical tasks

Empirical Studies of Parameter Efficient Methods for Large Language Models of Code and Knowledge Transfer to R

Super Tiny Language Models

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain

Is larger always better? Evaluating and prompting large language models for non-generative medical tasks

MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Large language models encode clinical knowledge

Language models scale reliably with over-training and on downstream tasks

Model development for bespoke large language models for digital triage assistance in mental health care

Fine-Tuning Large Language Models to Enhance Programmatic Assessment in Graduate Medical Education

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Lightweight transformers for clinical natural language processing

SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees

Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care

Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

Achieving Peak Performance for Large Language Models: A Systematic Review

A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making