BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Jinhyuk Lee,Wonjin Yoon,Sungdong Kim,Donghyeon Kim,Sunkyu Kim,Chan Ho So,Jaewoo Kang

DOI: https://doi.org/10.1093/bioinformatics/btz682

2019-10-18

Abstract:Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. We make the pre-trained weights of BioBERT freely available at <a class="link-external link-https" href="https://github.com/naver/biobert-pretrained" rel="external noopener nofollow">this https URL</a>, and the source code for fine-tuning BioBERT available at <a class="link-external link-https" href="https://github.com/dmis-lab/biobert" rel="external noopener nofollow">this https URL</a>.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: With the rapid growth of the amount of biomedical literature, directly applying the latest progress in the field of natural language processing (NLP) to biomedical text mining has poor results. Specifically, there are significant differences between the word distributions in the general domain and those in the biomedical domain, resulting in unsatisfactory performance in biomedical text mining tasks.

To solve this problem, the author proposes a pre - trained language model for the biomedical field - BioBERT. BioBERT is improved based on the BERT model. By pre - training on a large - scale biomedical corpus, its performance in biomedical text mining tasks is improved.

### Main contributions:
1. **Propose BioBERT for the first time**: A pre - trained language model specifically for biomedical text mining, which is based on the BERT architecture and pre - trained on a biomedical corpus.
2. **Significantly improve performance**: BioBERT has achieved better results than the existing state - of - the - art models in multiple biomedical text mining tasks, especially in named entity recognition (NER), relation extraction (RE) and question answering (QA) tasks.
3. **Open - source resources**: The author provides the pre - trained BioBERT weights, pre - processed data sets, and the source code for fine - tuning BioBERT, which is convenient for other researchers to use and conduct further research.

### Specific improvements:
- **Pre - training strategy**: BioBERT is not only pre - trained on general corpora (such as Wikipedia and BooksCorpus), but also undergoes additional pre - training on biomedical corpora (such as PubMed abstracts and PMC full - text articles).
- **Fine - tuning effect**: Through fine - tuning, BioBERT has obtained performance improvements of 0.62%, 2.80% and 12.24% in biomedical named entity recognition, relation extraction and question answering tasks respectively.

### Conclusion:
By pre - training BERT on a biomedical corpus, BioBERT can better understand and process complex biomedical texts, thereby significantly improving the performance in biomedical text mining tasks. This provides strong support for future biomedical natural language processing research.