Multimodal LLMs for health grounded in individual-specific data

Anastasiya Belyaeva,Justin Cosentino,Farhad Hormozdiari,Krish Eswaran,Shravya Shetty,Greg Corrado,Andrew Carroll,Cory Y. McLean,Nicholas A. Furlotte

2023-07-20

Abstract:Foundation large language models (LLMs) have shown an impressive ability to solve tasks across a wide range of fields including health. To effectively solve personalized health tasks, LLMs need the ability to ingest a diversity of data modalities that are relevant to an individual's health status. In this paper, we take a step towards creating multimodal LLMs for health that are grounded in individual-specific data by developing a framework (HeLM: Health Large Language Model for Multimodal Understanding) that enables LLMs to use high-dimensional clinical modalities to estimate underlying disease risk. HeLM encodes complex data modalities by learning an encoder that maps them into the LLM's token embedding space and for simple modalities like tabular data by serializing the data into text. Using data from the UK Biobank, we show that HeLM can effectively use demographic and clinical features in addition to high-dimensional time-series data to estimate disease risk. For example, HeLM achieves an AUROC of 0.75 for asthma prediction when combining tabular and spirogram data modalities compared with 0.49 when only using tabular data. Overall, we find that HeLM outperforms or performs at parity with classical machine learning approaches across a selection of eight binary traits. Furthermore, we investigate the downstream uses of this model such as its generalizability to out-of-distribution traits and its ability to power conversations around individual health and wellness.

Quantitative Methods,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the issue of how to effectively assess disease risk by leveraging large language models (LLMs) in combination with individual-specific multimodal data. Specifically, the paper proposes a framework called HeLM (Health Large Language Model for Multimodal Understanding), which can encode various complex data modalities (such as clinical data, high-dimensional time series data, etc.) into the word embedding space of LLMs, enabling the LLM to assess an individual's disease risk based on these data.

The main contributions of the paper include:
1. **Proposing the HeLM framework**: By learning to map non-textual modality data into the word embedding space of LLMs, the framework allows LLMs to process and understand health-related data from different modalities.
2. **Performance validation**: Using the UK Biobank dataset, the performance of HeLM was validated across multiple disease classification tasks, showing that HeLM outperformed or was comparable to traditional machine learning methods (such as logistic regression and XGBoost) in certain tasks.
3. **Application of multimodal data**: The paper particularly explores the effectiveness of HeLM in assessing asthma risk by combining tabular data and pulmonary function test data (such as spirometry curves).

In summary, the paper attempts to improve the accuracy of disease risk assessment by developing the HeLM framework, thereby providing new technological means for personalized healthcare through the use of multimodal individual-specific data.