Abstract:Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 2313.

Deep learning based feature-level integration of multi-omics data for breast cancer patients survival analysis

Integrating multi-omics data by learning modality invariant representations for improved prediction of overall survival of cancer

Deep learning assisted multi-omics integration for survival and drug-response prediction in breast cancer

Integrative survival analysis of breast cancer with gene expression and DNA methylation data

Pancancer survival prediction using a deep learning architecture with multimodal representation and integration

Strategies to integrate multi-omics data for patient survival prediction

Enhancing Breast Cancer Survival Prognosis Through Omic and Non-Omic Data Integration

Deep Learning-Based Multiomics Integration Model for Predicting Axillary Lymph Node Metastasis in Breast Cancer.

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Integrating multi-omics data through deep learning for accurate cancer prognosis prediction

Integrating Somatic Mutations for Breast Cancer Survival Prediction Using Machine Learning Methods

Integrative data analysis of multi-platform cancer data with a multimodal deep learning approach

Interpretable meta-learning of multi-omics data for survival analysis and pathway enrichment

Improving prediction performance of colon cancer prognosis based on the integration of clinical and multi-omics data

Predicting censored survival data based on the interactions between meta-dimensional omics data in breast cancer

Integration of Multimodal Data from Disparate Sources for Identifying Disease Subtypes

Deep learning-based multi-modal data integration enhancing breast cancer disease-free survival prediction

Abstract 5395: Improving survival prediction using flexible late fusion machine learning framework for multi-omics data integration

Integration of multi-omics data for survival prediction of lung adenocarcinoma

Enhancing Non-Small Cell Lung Cancer Survival Prediction through Multi-Omics Integration Using Graph Attention Network

SALMON: Survival Analysis Learning with Multi-Omics Neural Networks on Breast Cancer