Abstract:Background: Although electronic health record systems have facilitated clinical documentation in health care, they have also introduced new challenges, such as the proliferation of redundant information through the use of copy and paste commands or templates. One approach to trimming down bloated clinical documentation and improving clinical summarization is to identify highly similar text snippets with the goal of removing such text. Objective: We developed a natural language processing system for the task of assessing clinical semantic textual similarity. The system assigns scores to pairs of clinical text snippets based on their clinical semantic similarity. Methods: We leveraged recent advances in natural language processing and graph representation learning to create a model that combines linguistic and domain knowledge information from the MedSTS data set to assess clinical semantic textual similarity. We used bidirectional encoder representation from transformers (BERT)-based models as text encoders for the sentence pairs in the data set and graph convolutional networks (GCNs) as graph encoders for corresponding concept graphs that were constructed based on the sentences. We also explored techniques, including data augmentation, ensembling, and knowledge distillation, to improve the model's performance, as measured by the Pearson correlation coefficient (r). Results: Fine-tuning the BERT_base and ClinicalBERT models on the MedSTS data set provided a strong baseline (Pearson correlation coefficients: 0.842 and 0.848, respectively) compared to those of the previous year's submissions. Our data augmentation techniques yielded moderate gains in performance, and adding a GCN-based graph encoder to incorporate the concept graphs also boosted performance, especially when the node features were initialized with pretrained knowledge graph embeddings of the concepts (r=0.868). As expected, ensembling improved performance, and performing multisource ensembling by using different language model variants, conducting knowledge distillation with the multisource ensemble model, and taking a final ensemble of the distilled models further improved the system's performance (Pearson correlation coefficients: 0.875, 0.878, and 0.882, respectively). Conclusions: This study presents a system for the MedSTS clinical semantic textual similarity benchmark task, which was created by combining BERT-based text encoders and GCN-based graph encoders in order to incorporate domain knowledge into the natural language processing pipeline. We also experimented with other techniques involving data augmentation, pretrained concept embeddings, ensembling, and knowledge distillation to further increase our system's performance. Although the task and its benchmark data set are in the early stages of development, this study, as well as the results of the competition, demonstrates the potential of modern language model-based systems to detect redundant information in clinical notes.

Concept Graphs: A Novel Approach for Textual Analysis of Medical Documents

From Text to Topics in Healthcare Records: An Unsupervised Graph Partitioning Methodology

Abstracting Deep Neural Networks into Concept Graphs for Concept Level Interpretability

GRAPH: A domain ontology-driven semantic graph auto extraction system

Concept Graph Neural Networks for Surgical Video Understanding

From Free Text to Clusters of Content in Health Records: An Unsupervised Graph Partitioning Approach

Semantic Health Knowledge Graph: Semantic Integration of Heterogeneous Medical Knowledge and Services

A knowledge-driven approach to biomedical document conceptualization

Graph AI in Medicine

Incorporating Domain Knowledge Into Language Models by Using Graph Convolutional Networks for Assessing Semantic Textual Similarity: Model Development and Performance Comparison

CTGA: Graph-based Biomedical Literature Search

Natural Language Processing and Graph Theory: Making Sense of Imaging Records in a Novel Representation Frame

Bringing Semantic Structures to User Intent Detection in Online Medical Queries

Identify diabetic retinopathy-related clinical concepts and their attributes using transformer-based natural language processing methods

Discovering Related Clinical Concepts Using Large Amounts of Clinical Notes

Q-Map: Clinical Concept Mining from Clinical Documents

MedGraph: An experimental semantic information retrieval method using knowledge graph embedding for the biomedical citations indexed in PubMed

A Concept-Centered Hypertext Approach to Case-Based Retrieval

Graph Artificial Intelligence in Medicine

Using graph transformation algorithms to generate natural language equivalents of icons expressing medical concepts

Natural Language Processing and Semantical Representation of Medical Texts