Abstract:Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

Deep scaled dot-product attention based domain adaptation model for biomedical question answering

Hierarchical Multi-layer Transfer Learning Model for Biomedical Question Answering

Neural Domain Adaptation for Biomedical Question Answering

Improving Biomedical Question Answering by Data Augmentation and Model Weighting

Hierarchical Question-Aware Context Learning with Augmented Data for Biomedical Question Answering.

Dual Model Weighting Strategy and Data Augmentation in Biomedical Question Answering.

Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering

Named Entity Aware Transfer Learning for Biomedical Factoid Question Answering

External features enriched model for biomedical question answering

A Deep Transfer Learning Method for Medical Question Matching.

Optimized Biomedical Question-Answering Services with LLM and Multi-BERT Integration

Adapting and evaluating a deep learning language model for clinical why-question answering

Multi-task Transfer Learning for Biomedical Machine Reading Comprehension.

Adversarial Transfer Learning for Biomedical Named Entity Recognition

Advancing Domain Adaptation of BERT by Learning Domain Term Semantics.

Medical Question Retrieval Based on Siamese Neural Network and Transfer Learning Method.

Exploration of Attention Mechanism-Enhanced Deep Learning Models in the Mining of Medical Textual Data

Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Biomedical-domain Pre-Trained Language Model for Extractive Summarization