Abstract:Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 2313.

Multimodal Data Integration Using Deep Learning Predicts Overall Survival of Patients with Glioma

Deep Learning for the Prediction of the Survival of Midline Diffuse Glioma with an H3K27M Alteration

Predicting glioblastoma molecular subtypes and prognosis with a multimodal model integrating convolutional neural network, radiomics, and semantics

Multi-Channel 3D Deep Feature Learning for Survival Time Prediction of Brain Tumor Patients Using Multi-Modal Neuroimages

Survival Prediction of LGG Patients Based on Deep Learning of MRI Data and Genomic Data Integration

NIMG-60. PREDICTING OVERALL SURVIVAL IN GLIOBLASTOMA USING HISTOPATHOLOGY VIA AN END-TO-END DEEP LEARNING PIPELINE: A LARGE MULTI-COHORT STUDY

Prediction of overall survival and molecular markers in gliomas via analysis of digital pathology images using deep learning

An online survival predictor in glioma patients using machine learning based on WHO CNS5 data

Pre-operative Overall Survival Time Prediction for Glioblastoma Patients Using Deep Learning on Both Imaging Phenotype and Genotype

Non-Invasive Prediction of Survival Time of Midline Glioma Patients Using Machine Learning on Multiparametric MRI Radiomics Features

Multimodal Deep Learning-Based Prognostication in Glioma Patients: A Systematic Review

A Deep Learning Model with Radiomics Analysis Integration for Glioblastoma Post-Resection Survival Prediction

Exploring the prognostic value and biological pathways of transcriptomics and radiomics patterns in glioblastoma multiforme

Multiple Survival Outcome Prediction of Glioblastoma Patients Based on Multiparametric MRI

Biologically interpretable multi-task deep learning pipeline predicts molecular alterations, grade, and prognosis in glioma patients

Ensemble Learning for Glioma Patients Overall Survival Prediction Using Pre-Operative MRIs

Pancancer survival prediction using a deep learning architecture with multimodal representation and integration

Overall Survival Prediction for Gliomas Using a Novel Compound Approach.

Machine learning classifiers for predicting 3-year progression-free survival and overall survival in patients with gliomas after surgery.

Multimodal deep learning to predict prognosis in adult and pediatric brain tumors

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images