MultiMed: Massively Multimodal and Multitask Medical Understanding

Shentong Mo,Paul Pu Liang
2024-08-23
Abstract:Biomedical data is inherently multimodal, consisting of electronic health records, medical imaging, digital pathology, genome sequencing, wearable sensors, and more. The application of artificial intelligence tools to these multifaceted sensing technologies has the potential to revolutionize the prognosis, diagnosis, and management of human health and disease. However, current approaches to biomedical AI typically only train and evaluate with one or a small set of medical modalities and tasks. This limitation hampers the development of comprehensive tools that can leverage the rich interconnected information across many heterogeneous biomedical sensors. To address this challenge, we present MultiMed, a benchmark designed to evaluate and enable large-scale learning across a wide spectrum of medical modalities and tasks. MultiMed consists of 2.56 million samples across ten medical modalities such as medical reports, pathology, genomics, and protein data, and is structured into eleven challenging tasks, including disease prognosis, protein structure prediction, and medical question answering. Using MultiMed, we conduct comprehensive experiments benchmarking state-of-the-art unimodal, multimodal, and multitask models. Our analysis highlights the advantages of training large-scale medical models across many related modalities and tasks. Moreover, MultiMed enables studies of generalization across related medical concepts, robustness to real-world noisy data and distribution shifts, and novel modality combinations to improve prediction performance. MultiMed will be publicly available and regularly updated and welcomes inputs from the community.
Machine Learning,Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Multimedia
What problem does this paper attempt to address?
The paper attempts to address the issue that existing artificial intelligence models in the biomedical field are typically trained and evaluated on a single or a small number of medical modalities and tasks, failing to fully utilize the rich interconnection information between various heterogeneous biomedical sensors. To overcome this limitation, the researchers proposed a large-scale benchmark dataset named MultiMed, aimed at evaluating and promoting large-scale learning across a wide range of medical modalities and tasks. MultiMed contains over 2.5 million samples, covering 10 different medical modalities such as medical reports, pathology, genomics, and proteomics data, and is structured into 11 challenging tasks, including disease prognosis, protein structure prediction, and medical question answering. Through MultiMed, the researchers conducted extensive experiments to benchmark state-of-the-art single-modal, multi-modal, and multi-task models. The analysis indicates that the advantages of training large-scale medical models across multiple related modalities and tasks are significant. Additionally, MultiMed enables researchers to study the generalization capabilities between related medical concepts, robustness to real-world noisy data and distribution shifts, and novel modality combinations to improve prediction performance.