Abstract:Background: The electronic health record (EHR) contains a wealth of medical information. An organized EHR can greatly help doctors treat patients. In some cases, only limited patient information is collected to help doctors make treatment decisions. Because EHRs can serve as a reference for this limited information, doctors' treatment capabilities can be enhanced. Natural language processing and deep learning methods can help organize and translate EHR information into medical knowledge and experience. Objective: In this study, we aimed to create a model to extract concept embeddings from EHRs for disease pattern retrieval and further classification tasks. Methods: We collected 1,040,989 emergency department visits from the National Taiwan University Hospital Integrated Medical Database and 305,897 samples from the National Hospital and Ambulatory Medical Care Survey Emergency Department data. After data cleansing and preprocessing, the data sets were divided into training, validation, and test sets. We proposed a Transformer-based model to embed EHRs and used Bidirectional Encoder Representations from Transformers (BERT) to extract features from free text and concatenate features with structural data as input to our proposed model. Then, Deep InfoMax (DIM) and Simple Contrastive Learning of Visual Representations (SimCLR) were used for the unsupervised embedding of the disease concept. The pretrained disease concept-embedding model, named EDisease, was further finetuned to adapt to the critical care outcome prediction task. We evaluated the performance of embedding using t-distributed stochastic neighbor embedding (t-SNE) to perform dimension reduction for visualization. The performance of the finetuned predictive model was evaluated against published models using the area under the receiver operating characteristic (AUROC). Results: The performance of our model on the outcome prediction had the highest AUROC of 0.876. In the ablation study, the use of a smaller data set or fewer unsupervised methods for pretraining deteriorated the prediction performance. The AUROCs were 0.857, 0.870, and 0.868 for the model without pretraining, the model pretrained by only SimCLR, and the model pretrained by only DIM, respectively. On the smaller finetuning set, the AUROC was 0.815 for the proposed model. Conclusions: Through contrastive learning methods, disease concepts can be embedded meaningfully. Moreover, these methods can be used for disease retrieval tasks to enhance clinical practice capabilities. The disease concept model is also suitable as a pretrained model for subsequent prediction tasks.

Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction

Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

DeepHealth: Deep Representation Learning with Autoencoders for Healthcare Prediction

BEHRT: Transformer for Electronic Health Records

Med-BERT: A Pre-Training Framework for Medical Records Named Entity Recognition

DiLBERT: Cheap Embeddings for Disease Related Medical NLP

Disease Concept-Embedding Based on the Self-Supervised Method for Medical Information Extraction from Electronic Health Records and Disease Retrieval: Algorithm Development and Validation Study

AD-BERT: Using pre-trained language model to predict the progression from mild cognitive impairment to Alzheimer's disease

CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks

Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks

AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease

TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records

Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records

Task-Oriented Predictive (Top)-BERT: Novel Approach for Predicting Diabetic Complications Using a Single-Center EHR Data

A general approach for improving deep learning-based medical relation extraction using a pre-trained model and fine-tuning

al-BERT: a semi-supervised denoising technique for disease prediction

Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data

Prediction of Chronic Kidney Disease Risk Using Multimodal Data

Enhancing the prediction of disease outcomes using electronic health records and pretrained deep learning models

Electronic Health Records-Based Data-Driven Diabetes Knowledge Unveiling and Risk Prognosis