Multi-Modal Contrastive Learning for Online Clinical Time-Series Applications

Fabian Baldenweg,Manuel Burger,Gunnar Rätsch,Rita Kuznetsova

2024-03-27

Abstract:Electronic Health Record (EHR) datasets from Intensive Care Units (ICU) contain a diverse set of data modalities. While prior works have successfully leveraged multiple modalities in supervised settings, we apply advanced self-supervised multi-modal contrastive learning techniques to ICU data, specifically focusing on clinical notes and time-series for clinically relevant online prediction tasks. We introduce a loss function Multi-Modal Neighborhood Contrastive Loss (MM-NCL), a soft neighborhood function, and showcase the excellent linear probe and zero-shot performance of our approach.

Machine Learning

What problem does this paper attempt to address?

This paper mainly explores how to improve the performance of online clinical prediction tasks for Intensive Care Unit (ICU) Electronic Health Record (EHR) data using Multi-Modal Contrastive Learning. The focus of the research is on clinical notes and time series data. The authors propose a new loss function - Multi-Modal Neighborhood Contrastive Learning (MM-NCL), as well as a soft neighborhood function to enhance the model's performance in linear exploration and zero-shot tasks.

The paper mentions that although previous work has successfully utilized various modalities of data in a supervised learning environment, these methods often require separate training for each task and rely on a large amount of annotated data. Therefore, they adopt a self-supervised approach to learn task-agnostic representations, reducing the reliance on annotated data. Contrastive learning has been proven effective in multi-modal representations of images and text, achieving strong zero-shot classification performance without task-specific training.

The main contributions of the paper are as follows:

1. Proposing the MM-NCL loss function and soft neighborhood function for multi-modal contrastive learning of clinical notes and medical time series data.
2. Demonstrating the excellent linear exploration and zero-shot performance of this method in online prediction tasks for in-hospital mortality and deterioration of patient condition.
3. In the prediction task of deterioration of patient condition, the authors claim that their results represent the best benchmark for zero-shot prediction.

In the experimental section, the authors used the MIMIC-III dataset and compared it with other supervised and self-supervised methods, demonstrating that their method outperforms existing techniques in certain tasks, especially in cases where data annotation is limited and zero-shot prediction performance is more prominent. Additionally, they analyzed the impact of different types of clinical notes on the model's performance.