Abstract:Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 2313.

Pathformer: a Biological Pathway Informed Transformer for Disease Diagnosis and Prognosis Using Multi-Omics Data.

Pathformer: biological pathway informed Transformer model integrating multi-modal data

Transformer-based deep learning integrates multi-omic data with cancer pathways

Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction

Tracing unknown tumor origins with a biological-pathway-based transformer model

Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis

PathMethy: an interpretable AI framework for cancer origin tracing based on DNA methylation

Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction

Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction

DeePathology: Deep Multi-Task Learning for Inferring Molecular Pathology from Cancer Transcriptome

Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview

PG-TFNet: Transformer-based Fusion Network Integrating Pathological Images and Genomic Data for Cancer Survival Analysis

Pathway-Based Personalized Analysis of Pan-Cancer Transcriptomic Data

Surformer: an interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images

TransSurv: Transformer-Based Survival Analysis Model Integrating Histopathological Images and Genomic Data for Colorectal Cancer.

ConvPath: A Software Tool for Lung Adenocarcinoma Digital Pathological Image Analysis Aided by Convolutional Neural Network

PathFinder: a novel graph transformer model to infer multi-cell intra- and inter-cellular signaling pathways and communications

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Exploit Spatially Resolved Transcriptomic Data to Infer Cellular Features from Pathology Imaging Data}

Pathway-enhanced Transformer-based robust model for quantifying cell types of origin of cell-free transcriptome

Highly Accurate Disease Diagnosis and Highly Reproducible Biomarker Identification with PathFormer