Abstract:Background: Semantic textual similarity (STS) is one of the fundamental tasks in natural language processing (NLP). Many shared tasks and corpora for STS have been organized and curated in the general English domain; however, such resources are limited in the biomedical domain. In 2019, the National NLP Clinical Challenges (n2c2) challenge developed a comprehensive clinical STS dataset and organized a community effort to solicit state-of-the-art solutions for clinical STS. Objective: This study presents our transformer-based clinical STS models developed during this challenge as well as new models we explored after the challenge. This project is part of the 2019 n2c2/Open Health NLP shared task on clinical STS. Methods: In this study, we explored 3 transformer-based models for clinical STS: Bidirectional Encoder Representations from Transformers (BERT), XLNet, and Robustly optimized BERT approach (RoBERTa). We examined transformer models pretrained using both general English text and clinical text. We also explored using a general English STS dataset as a supplementary corpus in addition to the clinical training set developed in this challenge. Furthermore, we investigated various ensemble methods to combine different transformer models. Results: Our best submission based on the XLNet model achieved the third-best performance (Pearson correlation of 0.8864) in this challenge. After the challenge, we further explored other transformer models and improved the performance to 0.9065 using a RoBERTa model, which outperformed the best-performing system developed in this challenge (Pearson correlation of 0.9010). Conclusions: This study demonstrated the efficiency of utilizing transformer-based models to measure semantic similarity for clinical text. Our models can be applied to clinical applications such as clinical text deduplication and summarization.

ECNUCS: Measuring Short Text Semantic Equivalence Using Multiple Similarity Measurements.

Measuring Short Text Semantic Similarity Using Multiple Measurements

ECNU: Using Traditional Similarity Measurements and Word Embedding for Semantic Textual Similarity Estimation.

ECNU: One Stone Two Birds: Ensemble of Heterogenous Measures for Semantic Relatedness and Textual Entailment

ECNU: Leveraging on Ensemble of Heterogeneous Features and Information Enrichment for Cross Level Semantic Similarity Estimation

BIT at SemEval-2017 Task 1: Using Semantic Information Space to Evaluate Semantic Textual Similarity.

Computing Semantic Text Similarity Using Rich Features.

A Semantic Textual Similarity Measurement Model Based on the Syntactic-Semantic Representation

Tiantianzhu7:System Description of Semantic Textual Similarity (STS) in the SemEval-2012 (task 6)

Measurement of Semantic Textual Similarity in Clinical Texts: Comparison of Transformer-Based Models

ECNUCS: Recognizing Cross-lingual Textual Entailment Using Multiple Text Similarity and Text Difference Measures.

Attentive Siamese LSTM Network for Semantic Textual Similarity Measure

SXUCFN-Core: STS Models Integrating FrameNet Parsing Information.

Fine-grained Semantic Textual Similarity Measurement Via a Feature Separation Network

ECNU at SemEval-2017 Task 1: Leverage Kernel-based Traditional NLP Features and Neural Networks to Build a Universal Model for Multilingual and Cross-lingual Semantic Textual Similarity

C-STS: Conditional Semantic Textual Similarity

An Unsupervised Semantic Text Similarity Measurement Model in Resource-Limited Scenes

Fusing Syntax and Word Embedding Knowledge for Measuring Semantic Similarity

A survey on the techniques, applications, and performance of short text semantic similarity

BIT at SemEval-2016 Task 1: Sentence Similarity Based on Alignments and Vector with the Weight of Information Content.

PolyUCOMP-CORE_TYPED: Computing Semantic Textual Similarity Using Overlapped Senses.