Abstract:Unsupervised pre-trained language models (PLMs) have boosted the development of effective biomedical text mining models. But the biomedical texts contain a huge number of long-tail concepts and terminologies, which makes further pre-training on biomedical corpora relatively expensive (more biomedical corpora and more pre-training steps are needed). Nonetheless, this problem receives less attention in recent studies. In Chinese biomedical text, concepts and terminologies consist of Chinese characters, and Chinese characters are often composed of sub-character components which are also semantically informative; thus in order to enhance the semantics of biomedical concepts and terminologies, the use of a Chinese character’s component-level internal semantic information also appears to be reasonable.In this paper, we propose a novel hanzi-aware pre-trained language model for Chinese biomedical text mining, referred to as BioHanBERT (hanzi-aware BERT for Chinese biomedical text mining), utilizing the component-level internal semantic information of Chinese characters to enhance the semantics of Chinese biomedical concepts and terminologies, and thereby to reduce further pre-training costs. BioHanBERT first employs a Chinese character encoder to extract the component-level internal semantic feature of each Chinese character, and then fuse the character’s internal semantic feature and its contextual embedding extracted by BERT to enrich the representations of the concepts or terminologies containing the character. The results of extensive experiments show that our model is able to consistently outperform current state-of-the-art (SOTA) models in a wide range of Chinese biomedical natural language processing (NLP) tasks.

Fasthan: A BERT-based Multi-Task Toolkit for Chinese NLP.

Fasthan: A BERT-based Joint Many-Task Toolkit for Chinese NLP

Chinese Text Classification Using BERT and Flat-Lattice Transformer.

Development and Evaluation of Task-Specific NLP Framework in China.

FudanNLP: A Toolkit for Chinese Natural Language Processing.

BioHanBERT - A Hanzi-aware Pre-trained Language Model for Chinese Biomedical Text Mining.

FudanNLP: A Toolkit for Chinese Natural Language Processing with Online Learning Algorithms

Fast and Accurate FSA System Using ELBERT: An Efficient and Lightweight BERT

A More Efficient Chinese Named Entity Recognition base on BERT and Syntactic Analysis

A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsing

Fast Neural Chinese Named Entity Recognition with Multi-head Self-attention

Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT

Unified Mandarin TTS Front-end Based on Distilled BERT Model

PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing

TCBERT: A Technical Report for Chinese Topic Classification BERT

A Flexible Multi-Task Model for BERT Serving

EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing

OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Services

NEZHA: Neural Contextualized Representation for Chinese Language Understanding

Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning

Unified Framework of Performing Chinese Word Segmentation and Part-Of-Speech Tagging