Abstract:Background: Semantic textual similarity (STS) is one of the fundamental tasks in natural language processing (NLP). Many shared tasks and corpora for STS have been organized and curated in the general English domain; however, such resources are limited in the biomedical domain. In 2019, the National NLP Clinical Challenges (n2c2) challenge developed a comprehensive clinical STS dataset and organized a community effort to solicit state-of-the-art solutions for clinical STS. Objective: This study presents our transformer-based clinical STS models developed during this challenge as well as new models we explored after the challenge. This project is part of the 2019 n2c2/Open Health NLP shared task on clinical STS. Methods: In this study, we explored 3 transformer-based models for clinical STS: Bidirectional Encoder Representations from Transformers (BERT), XLNet, and Robustly optimized BERT approach (RoBERTa). We examined transformer models pretrained using both general English text and clinical text. We also explored using a general English STS dataset as a supplementary corpus in addition to the clinical training set developed in this challenge. Furthermore, we investigated various ensemble methods to combine different transformer models. Results: Our best submission based on the XLNet model achieved the third-best performance (Pearson correlation of 0.8864) in this challenge. After the challenge, we further explored other transformer models and improved the performance to 0.9065 using a RoBERTa model, which outperformed the best-performing system developed in this challenge (Pearson correlation of 0.9010). Conclusions: This study demonstrated the efficiency of utilizing transformer-based models to measure semantic similarity for clinical text. Our models can be applied to clinical applications such as clinical text deduplication and summarization.

ISCAS at SemEval-2020 Task 5: Pre-trained Transformers for Counterfactual Statement Modeling

SemiCVT: Semi-Supervised Convolutional Vision Transformer for Semantic Segmentation

BUT-FIT at SemEval-2020 Task 5: Automatic detection of counterfactual statements with deep pre-trained language representation models

CS-NET at SemEval-2020 Task 4: Siamese BERT for ComVE

Measurement of Semantic Textual Similarity in Clinical Texts: Comparison of Transformer-Based Models

Predicting semantic category of answers for question answering systems using transformers: a transfer learning approach

AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles

problemConquero at SemEval-2020 Task 12: Transformer and Soft label-based approaches

5q032e@SMM4H'22: Transformer-based classification of premise in tweets related to COVID-19

Condition Aware and Revise Transformer for Question Answering.

Unsupervised Evaluation for Question Answering with Transformers

A Cognitive Study on Semantic Similarity Analysis of Large Corpora: A Transformer-based Approach

Empowering Language Understanding with Counterfactual Reasoning

ICL00 at SemEval-2016 Task 3: Translation-Based Method for CQA System

Correction of Faulty Background Knowledge based on Condition Aware and Revise Transformer for Question Answering

SemEval-2020 Task 4: Commonsense Validation and Explanation.

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning

Survey of transformers and towards ensemble learning using transformers for natural language processing

Positional Attention Guided Transformer-Like Architecture for Visual Question Answering

Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task

Learning to generate and evaluate fact-checking explanations with transformers