Abstract:Text embeddings convert textual information into numerical representations, enabling machines to perform semantic tasks like information retrieval. Despite its potential, the application of text embeddings in healthcare is underexplored in part due to a lack of benchmarking studies using biomedical data. This study provides a flexible framework for benchmarking embedding models to identify those most effective for healthcare-related semantic tasks. We selected thirty embedding models from the multilingual text embedding benchmarks (MTEB) Hugging Face resource, of various parameter sizes and architectures. Models were tested with real-world semantic retrieval medical tasks on (1) PubMed abstracts, (2) synthetic Electronic Health Records (EHRs) generated by the Llama-3-70b model, (3) real-world patient data from the Mount Sinai Health System, and the (4) MIMIC IV database. Tasks were split into Short Tasks, involving brief text pair interactions such as triage notes and chief complaints, and Long Tasks, which required processing extended documentation such as progress notes and history & physical notes. We assessed models by correlating their performance with data integrity levels, ranging from 0% (fully mismatched pairs) to 100% (perfectly matched pairs), using Spearman correlation. Additionally, we examined correlations between the average Spearman scores across tasks and two MTEB leaderboard benchmarks: the overall recorded average and the average Semantic Textual Similarity (STS) score. We evaluated 30 embedding models across seven clinical tasks (each involving 2,000 text pairs), across five levels of data integrity, totaling 2.1 million comparisons. Some models performed consistently well, while models based on Mistral-7b excelled in long-context tasks. NV-Embed-v1, despite being top performer in short tasks, did not perform as well in long tasks. Our average task performance score (ATPS) correlated better with the MTEB STS score (0.73) than with MTEB average score (0.67). The suggested framework is flexible, scalable and resistant to the risk of models overfitting on published benchmarks. Adopting this method can improve embedding technologies in healthcare.

Profiling Clinical Researchers Effectively using Embeddings and Clustering

Patient Clustering via Integrated Profiling of Clinical and Digital Data

A Clustering Algorithm Based on Document Embedding to Identify Clinical Note Templates

A Unified Approach to Researcher Profiling

A Study of Deep Learning Methods for De-identification of Clinical Notes at Cross Institute Settings.

Medical Profile Model: Scientific and Practical Applications in Healthcare

WellFactor: Patient Profiling using Integrative Embedding of Healthcare Data

Enabling scalable clinical interpretation of ML-based phenotypes using real world data

A Study of Neural Word Embeddings for Named Entity Recognition in Clinical Text.

An Empirical Study on Clustering Pretrained Embeddings: Is Deep Strictly Better?

A Scalable Framework for Benchmarking Embedding Models for Semantic Medical Tasks

Integrating Protein Sequence and Expression Level to Analysis Molecular Characterization of Breast Cancer Subtypes

Unified Clinical Vocabulary Embeddings for Advancing Precision Medicine

Mapping Unparalleled Clinical Professional and Consumer Languages with Embedding Alignment

Leveraging Pre-trained and Transformer-derived Embeddings from EHRs to Characterize Heterogeneity Across Alzheimer's Disease and Related Dementias

Expert Profile Identification From Community Detection on Author-Publication-Keyword Graph With Keyword Extraction

Towards User Profiling From Multiple Online Social Networks

K-Profiles: A Nonlinear Clustering Method For Pattern Detection In High Dimensional Data

Benchmarking Embedding Aggregation Methods in Computational Pathology: A Clinical Data Perspective

Accelerating discoveries in medicine using distributed vector representations of words

Snomed2Vec: Random Walk and Poincaré Embeddings of a Clinical Knowledge Base for Healthcare Analytics