Abstract:Highlights • Evaluated the impact of corpus heterogeneity on the generalizability of series NLP models, including CRF, BiLSTM-CRF and BERT based models. • Novel analysis of robustness of the machine learning models using permutation dataset. • Compared two strategies for transferring machine learning models between clinical institutes, i.e., i) direct transfer vs ii) continuous fine-tuning. Objective Transformer-based language models are prevailing in the clinical domain due to their excellent performance on clinical NLP tasks. The generalizability of those models is usually ignored during the model development process. This study evaluated the generalizability of CancerBERT, a Transformer-based clinical NLP model, along with classic machine learning models, i.e., conditional random field (CRF), bi-directional long short-term memory CRF (BiLSTM-CRF), across different clinical institutes through a breast cancer phenotype extraction task. Materials and Methods Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota and Mayo Clinic, and annotated following the same guideline. We developed three types of NLP models (i.e., CRF, BiLSTM-CRF and CancerBERT) to extract cancer phenotypes from clinical texts. We evaluated the generalizability of models on different test sets with different learning strategies (model transfer vs locally trained). The entity coverage score was assessed with their association with the model performances. Results We manually annotated 200 and 161 clinical documents at UMN and MC. The corpora of the two institutes were found to have higher similarity between the target entities than the overall corpora. The CancerBERT models obtained the best performances among the independent test sets from two clinical institutes and the permutation test set. The CancerBERT model developed in one institute and further fine-tuned in another institute achieved reasonable performance compared to the model developed on local data (micro-F1: 0.925 vs 0.932). Conclusions The results indicate the CancerBERT model has superior learning ability and generalizability among the three types of clinical NLP models for our named entity recognition task. It has the advantage to recognize complex entities, e.g., entities with different labels. Graphical abstract Download : Download high-res image (110KB) Download : Download full-size image

Using language models and ontology topology to perform semantic mapping of traits between biomedical datasets

A Cross‐Domain Ontology Semantic Representation Based on NCBI‐BlueBERT Embedding

Semantic computing for human phenotypes

Ontologizing health systems data at scale: making translational discovery a reality

Evaluating Biomedical BERT Models for Vocabulary Alignment at Scale in the UMLS Metathesaurus

A Natural Language Processing Approach to Support Biomedical Data Harmonization: Leveraging Large Language Models

OPA2Vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction

OntoSem: an Ontology Semantic Representation Methodology for Biomedical Domain

Comparing neural language models for medical concept representation and patient trajectory prediction

Ontological representation, classification and data-driven computing of phenotypes

Language-model-based patient embedding using electronic health records facilitates phenotyping, disease forecasting, and progression analysis

A Cross-institutional Evaluation on Breast Cancer Phenotyping NLP Algorithms on Electronic Health Records

Machine learning with biomedical ontologies

Linking rare and common disease vocabularies by mapping between the human phenotype ontology and phecodes

Automatic Structuring of Ontology Terms Based on Lexical Granularity and Machine Learning: Algorithm Development and Validation

Similarity-based Algorithms for Disease Terminology Mapping

sEHR-CE: Language modelling of structured EHR data for efficient and generalizable patient cohort expansion

A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts

Fully automatic extraction of morphological traits from the Web: utopia or reality?

Ontology-based Representations of Biological Entities