Abstract:Motivation: Cancer is a complex disease that results in a significant number of global fatalities. Treatment strategies can vary among patients, even if they have the same type of cancer. The application of precision medicine in cancer shows promise for treating different types of cancer, reducing healthcare expenses, and improving recovery rates. To achieve personalized cancer treatment, machine learning models have been developed to predict drug responses based on tumor and drug characteristics. However, current studies either focus on constructing homogeneous networks from single data source or heterogeneous networks from multiomics data. While multiomics data have shown potential in predicting drug responses in cancer cell lines, there is still a lack of research that effectively utilizes insights from different modalities. Furthermore, effectively utilizing the multimodal knowledge of cancer cell lines poses a challenge due to the heterogeneity inherent in these modalities. Results: To address these challenges, we introduce MMCL-CDR (Multimodal Contrastive Learning for Cancer Drug Responses), a multimodal approach for cancer drug response prediction that integrates copy number variation, gene expression, morphology images of cell lines, and chemical structure of drugs. The objective of MMCL-CDR is to align cancer cell lines across different data modalities by learning cell line representations from omic and image data, and combined with structural drug representations to enhance the prediction of cancer drug responses (CDR). We have carried out comprehensive experiments and show that our model significantly outperforms other state-of-the-art methods in CDR prediction. The experimental results also prove that the model can learn more accurate cell line representation by integrating multiomics and morphological data from cell lines, thereby improving the accuracy of CDR prediction. In addition, the ablation study and qualitative analysis also confirm the effectiveness of each part of our proposed model. Last but not least, MMCL-CDR opens up a new dimension for cancer drug response prediction through multimodal contrastive learning, pioneering a novel approach that integrates multiomics and multimodal drug and cell line modeling. Availability and implementation: MMCL-CDR is available at https://github.com/catly/MMCL-CDR.

MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems

Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data

A multimodal ensemble approach for clear cell renal cell carcinoma treatment outcome prediction

Multimodal Survival Ensemble Network: Integrating Genomic and Histopathological Insights for Enhanced Cancer Prognosis.

Multi-modality fusion using canonical correlation analysis methods: Application in breast cancer survival prediction from histology and genomics

MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images

Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction

Quantifying the advantage of multimodal data fusion for survival prediction in cancer patients

DRIM: Learning Disentangled Representations from Incomplete Multimodal Healthcare Data

Multi-modal Multi-instance Learning Using Weakly Correlated Histopathological Images and Tabular Clinical Information

Building Flexible, Scalable, and Machine Learning-ready Multimodal Oncology Datasets

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

MultiMed: Massively Multimodal and Multitask Medical Understanding

MedPix 2.0: A Comprehensive Multimodal Biomedical Dataset for Advanced AI Applications

MMCL-CDR: enhancing cancer drug response prediction with multi-omics and morphology images contrastive representation learning

MIF: Multi-Shot Interactive Fusion Model for Cancer Survival Prediction Using Pathological Image and Genomic Data

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

Multimodal CustOmics: A Unified and Interpretable Multi-Task Deep Learning Framework for Multimodal Integrative Data Analysis in Oncology

RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy

Cross-Modal Information Maximization for Medical Imaging: CMIM