Abstract:Background: Although electronic health records (EHRs) have been widely adopted in health care, effective use of EHR data is often limited because of redundant information in clinical notes introduced by the use of templates and copy-paste during note generation. Thus, it is imperative to develop solutions that can condense information while retaining its value. A step in this direction is measuring the semantic similarity between clinical text snippets. To address this problem, we participated in the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing Consortium (OHNLP) clinical semantic textual similarity (ClinicalSTS) shared task. Objective: This study aims to improve the performance and robustness of semantic textual similarity in the clinical domain by leveraging manually labeled data from related tasks and contextualized embeddings from pretrained transformer-based language models. Methods: The ClinicalSTS data set consists of 1642 pairs of deidentified clinical text snippets annotated in a continuous scale of 0-5, indicating degrees of semantic similarity. We developed an iterative intermediate training approach using multi-task learning (IIT-MTL), a multi-task training approach that employs iterative data set selection. We applied this process to bidirectional encoder representations from transformers on clinical text mining (ClinicalBERT), a pretrained domain-specific transformer-based language model, and fine-tuned the resulting model on the target ClinicalSTS task. We incrementally ensembled the output from applying IIT-MTL on ClinicalBERT with the output of other language models (bidirectional encoder representations from transformers for biomedical text mining [BioBERT], multi-task deep neural networks [MT-DNN], and robustly optimized BERT approach [RoBERTa]) and handcrafted features using regression-based learning algorithms. On the basis of these experiments, we adopted the top-performing configurations as our official submissions. Results: Our system ranked first out of 87 submitted systems in the 2019 n2c2/OHNLP ClinicalSTS challenge, achieving state-of-the-art results with a Pearson correlation coefficient of 0.9010. This winning system was an ensembled model leveraging the output of IIT-MTL on ClinicalBERT with BioBERT, MT-DNN, and handcrafted medication features. Conclusions: This study demonstrates that IIT-MTL is an effective way to leverage annotated data from related tasks to improve performance on a target task with a limited data set. This contribution opens new avenues of exploration for optimized data set selection to generate more robust and universal contextual representations of text in the clinical domain.

Detection of Medication Mentions and Medication Change Events in Clinical Notes Using Transformer-Based Models

Contextualized medication information extraction using Transformer-based deep learning architectures

Developing a natural language processing system using transformer-based models for adverse drug event detection in electronic health records

Extracting social determinants of health events with transformer-based multitask, multilabel named entity recognition

Extracting medication changes in clinical narratives using pre-trained language models

Improving Extraction of Clinical Event Contextual Properties from Electronic Health Records: A Comparative Study

Measurement of Semantic Textual Similarity in Clinical Texts: Comparison of Transformer-Based Models

Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting

Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Models

Transformer-based Named Entity Recognition for Parsing Clinical Trial Eligibility Criteria

A Study of Deep Learning Approaches for Medication and Adverse Drug Event Extraction from Clinical Text.

ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

GNTeam at 2018 n2c2: Feature-augmented BiLSTM-CRF for drug-related entity recognition in hospital discharge summaries

Identification of Semantically Similar Sentences in Clinical Notes: Iterative Intermediate Training Using Multi-Task Learning

BCH-NLP at BioCreative VII Track 3: medications detection in tweets using transformer networks and multi-task learning

CACER: Clinical concept Annotations for Cancer Events and Relations

Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation

Applying and Improving a Publicly Available Medication NER Pipeline in a Clinical Cancer EMR

Clinical Named Entity Recognition using Contextualized Token Representations

TocBERT: Medical Document Structure Extraction Using Bidirectional Transformers

Automatic Extraction of Medication Names in Tweets as Named Entity Recognition