Abstract:Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 2313.

Deep learning-based survival prediction for multiple cancer types using histopathology images

Interpretable Survival Prediction for Colorectal Cancer using Deep Learning

Survival Prediction of Stomach Cancer Using Expression Data and Deep Learning Models with Histopathological Images

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images

Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning

Histopathology images-based deep learning prediction of prognosis and therapeutic response in small cell lung cancer

Diagnosis and Survival Prediction of Colorectal Cancer Using Deep Learning from the Histopathological HE Images

Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study

Deep Learning for Survival Analysis in Breast Cancer with Whole Slide Image Data

Deep learning-based multi-modal data integration enhancing breast cancer disease-free survival prediction

Prediction of overall survival and molecular markers in gliomas via analysis of digital pathology images using deep learning

Prediction of Disease-Free Survival in Breast Cancer using Deep Learning with Ultrasound and Mammography: A Multicenter Study

Classifying Subtypes and Predicting Survival of Renal Cell Carcinoma Using Histopathology Image-Based Deep Learning

Abstract 6175: Deep learning-based survival prediction for hepatocellular carcinoma using whole slide images

From slides to insights: Harnessing deep learning for prognostic survival prediction in human colorectal cancer histology

ICSDA: a multi-modal deep learning model to predict breast cancer recurrence and metastasis risk by integrating pathological, clinical and gene expression data

Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images

Biology-guided deep learning predicts prognosis and cancer immunotherapy response

Cancer drug sensitivity prediction from routine histology images

DeepCG: A cell graph model for predicting prognosis in lung adenocarcinoma