Contrastive Learning on Multimodal Analysis of Electronic Health Records

Tianxi Cai,Feiqing Huang,Ryumei Nakada,Linjun Zhang,Doudou Zhou

2024-03-22

Abstract:Electronic health record (EHR) systems contain a wealth of multimodal clinical data including structured data like clinical codes and unstructured data such as clinical notes. However, many existing EHR-focused studies has traditionally either concentrated on an individual modality or merged different modalities in a rather rudimentary fashion. This approach often results in the perception of structured and unstructured data as separate entities, neglecting the inherent synergy between them. Specifically, the two important modalities contain clinically relevant, inextricably linked and complementary health information. A more complete picture of a patient's medical history is captured by the joint analysis of the two modalities of data. Despite the great success of multimodal contrastive learning on vision-language, its potential remains under-explored in the realm of multimodal EHR, particularly in terms of its theoretical understanding. To accommodate the statistical analysis of multimodal EHR data, in this paper, we propose a novel multimodal feature embedding generative model and design a multimodal contrastive loss to obtain the multimodal EHR feature representation. Our theoretical analysis demonstrates the effectiveness of multimodal learning compared to single-modality learning and connects the solution of the loss function to the singular value decomposition of a pointwise mutual information matrix. This connection paves the way for a privacy-preserving algorithm tailored for multimodal EHR feature representation learning. Simulation studies show that the proposed algorithm performs well under a variety of configurations. We further validate the clinical utility of the proposed algorithm in real-world EHR data.

Machine Learning

What problem does this paper attempt to address?

This paper aims to address the effective integration and analysis of structured and unstructured data in electronic health records (EHR), particularly by utilizing multimodal contrastive learning to extract more meaningful feature representations. Existing studies have mostly focused on single-modal data, while this paper proposes a novel multimodal feature embedding generation model and multimodal contrastive loss function to capture the synergistic effects between different modalities of data, thereby improving the accuracy and privacy protection of clinical data analysis. The superiority of this approach over single-modal learning methods in handling EHR data is demonstrated through theoretical analysis and empirical research.

Contrastive Learning on Multimodal Analysis of Electronic Health Records

Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision

Multimodal contrastive learning for diagnosing cardiovascular diseases from electrocardiography (ECG) signals and patient metadata

Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications

Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives

A decision support system in precision medicine: contrastive multimodal learning for patient stratification

Multimodal Data Matters: Language Model Pre-Training Over Structured and Unstructured Electronic Health Records

Multimodal risk prediction with physiological signals, medical images and clinical notes

Multi-gate Mixture of Multi-view Graph Contrastive Learning on Electronic Health Record

Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data

Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention

A Multimodal Contrastive Federated Learning for Digital Healthcare

How to Leverage Multimodal EHR Data for Better Medical Predictions?

M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

Two heads are better than one: Enhancing medical representations by pre-training over structured and unstructured electronic health records

Research on Multimodal Fusion of Temporal Electronic Medical Records

Learning Representations for Multilead Electrocardiograms From Morphology-Rhythm Contrast

Learning Representations for Multi-Lead Electrocardiograms from Morphology-Rhythm Contrast

M3Care: Learning with Missing Modalities in Multimodal Healthcare Data

Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisation

Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment