Abstract:Biomedical data is inherently multimodal, consisting of electronic health records, medical imaging, digital pathology, genome sequencing, wearable sensors, and more. The application of artificial intelligence tools to these multifaceted sensing technologies has the potential to revolutionize the prognosis, diagnosis, and management of human health and disease. However, current approaches to biomedical AI typically only train and evaluate with one or a small set of medical modalities and tasks. This limitation hampers the development of comprehensive tools that can leverage the rich interconnected information across many heterogeneous biomedical sensors. To address this challenge, we present MultiMed, a benchmark designed to evaluate and enable large-scale learning across a wide spectrum of medical modalities and tasks. MultiMed consists of 2.56 million samples across ten medical modalities such as medical reports, pathology, genomics, and protein data, and is structured into eleven challenging tasks, including disease prognosis, protein structure prediction, and medical question answering. Using MultiMed, we conduct comprehensive experiments benchmarking state-of-the-art unimodal, multimodal, and multitask models. Our analysis highlights the advantages of training large-scale medical models across many related modalities and tasks. Moreover, MultiMed enables studies of generalization across related medical concepts, robustness to real-world noisy data and distribution shifts, and novel modality combinations to improve prediction performance. MultiMed will be publicly available and regularly updated and welcomes inputs from the community.

Multimodal Single Cell Data Integration Challenge: Results and Lessons Learned

A sandbox for prediction and integration of DNA, RNA, and proteins in single cells

Graph Neural Networks for Multimodal Single-Cell Data Integration

Multimodal weakly supervised learning to identify disease-specific changes in single-cell atlases

Multimodal deep learning approaches for single-cell multi-omics data integration

Multigrate: single-cell multi-omic data integration

The multimodality cell segmentation challenge: toward universal solutions

Integrated analysis of multimodal single-cell data

Liam tackles complex multimodal single-cell data integration challenges

The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions

Multi-task benchmarking of single-cell multimodal omics integration methods

Joint Analysis of Single-Cell Data across Cohorts with Missing Modalities

Best practices for single-cell analysis across modalities

Integrating T-cell Receptor and Transcriptome for Large-Scale Single-Cell Immune Profiling Analysis

Regression-Based Analysis of Multimodal Single-Cell Data Integration Strategies

Multimodal CustOmics: A Unified and Interpretable Multi-Task Deep Learning Framework for Multimodal Integrative Data Analysis in Oncology

scMODAL: A general deep learning framework for comprehensive single-cell multi-omics data alignment with feature links

MIDAS: a deep generative model for mosaic integration and knowledge transfer of single-cell multimodal data

The performance of deep generative models for learning joint embeddings of single-cell multi-omics data

MultiMed: Massively Multimodal and Multitask Medical Understanding

Comprehensive View Embedding Learning for Single-Cell Multimodal Integration