Abstract:Motivation: The explosive increase of biomedical literature has made information extraction an increasingly important tool for biomedical research. A fundamental task is the recognition of biomedical named entities in text (BNER) such as genes/proteins, diseases and species. Recently, a domain-independent method based on deep learning and statistical word embeddings, called long short-term memory network-conditional random field (LSTM-CRF), has been shown to outperform state-of-the-art entity-specific BNER tools. However, this method is dependent on gold-standard corpora (GSCs) consisting of hand-labeled entities, which tend to be small but highly reliable. An alternative to GSCs are silver-standard corpora (SSCs), which are generated by harmonizing the annotations made by several automatic annotation systems. SSCs typically contain more noise than GSCs but have the advantage of containing many more training examples. Ideally, these corpora could be combined to achieve the benefits of both, which is an opportunity for transfer learning. In this work, we analyze to what extent transfer learning improves upon state-of-the-art results for BNER.Results: We demonstrate that transferring a deep neural network (DNN) trained on a large, noisy SSC to a smaller, but more reliable GSC significantly improves upon state-of-the-art results for BNER. Compared to a state-of-the-art baseline evaluated on 23 GSCs covering four different entity classes, transfer learning results in an average reduction in error of approximately 11%. We found transfer learning to be especially beneficial for target datasets with a small number of labels (approximately 6000 or less).Availability and implementation: Source code for the LSTM-CRF is available at https://github.com/Franck-Dernoncourt/NeuroNER/ and links to the corpora are available at https://github.com/BaderLab/Transfer-Learning-BNER-Bioinformatics-2018/.Supplementary information: Supplementary data are available at Bioinformatics online.

Efficient transfer learning for NLP with ELECTRA

Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity

ELECTRA is a Zero-Shot Learner, Too

Training ELECTRA Augmented with Multi-word Selection

Evaluating the structure of cognitive tasks with transfer learning

Efficient Post-Training Augmentation for Adaptive Inference in Heterogeneous and Distributed IoT Environments

Efficiently Reusing Old Models Across Languages via Transfer Learning

Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning

Energon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse Attention.

EEL: Efficiently Encoding Lattices for Reranking

Transfer Learning for Low-Resource Clinical Named Entity Recognition

ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring

No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder

$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning

Pre-training Language Model As a Multi-perspective Course Learner.

IrEne: Interpretable Energy Prediction for Transformers

Efficient Joint Learning for Clinical Named Entity Recognition and Relation Extraction Using Fourier Networks: A Use Case in Adverse Drug Events

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

Transfer learning for biomedical named entity recognition with neural networks

Towards scalable efficient on-device ASR with transfer learning