Abstract:Purpose: Cancer patients routinely undergo radiologic and pathologic evaluation for their diagnostic workup. These data modalities represent a valuable and readily available resource for developing new prognostic tools. Given their vast difference in spatial scales, effective methods to integrate the two modalities are currently lacking. Here, we aim to develop a multi-modal approach to integrate radiology and pathology images for predicting outcomes in cancer patients. Methods: We propose a multi-modal weakly-supervised deep learning framework to integrate radiology and pathology images for survival prediction. We first extract multi-scale features from whole-slide H&E-stained pathology images to characterize cellular and tissue phenotypes as well as spatial cellular organization. We then build a hierarchical co-attention transformer to effectively learn the multi-modal interactions between radiology and pathology image features. Finally, a multimodal risk score is derived by combining complementary information from two images modalities and clinical data for predicting outcome. We evaluate our approach in lung, gastric, and brain cancers with matched radiology and pathology images and clinical data available, each with separate training and external validation cohorts. Results: The multi-modal deep learning models achieved a reasonably high accuracy for predicting survival outcomes in the external validation cohorts (C-index range: 0.72-0.75 across three cancer types). The multi-modal prognostic models significantly improved upon single-modal approach based on radiology or pathology images or clinical data alone (C-index range: 0.53-0.71, P<0.01). The multi-modal deep learning models were significantly associated with disease-free survival and overall survival (hazard ratio range: 3.23-4.46, P<0.0001). In multivariable analyses, the models remained an independent prognostic factor (P<0.01) after adjusting for clinicopathological variables including cancer stage and tumor differentiation. Conclusions: The proposed multi-modal deep learning approach outperforms traditional methods for predicting survival outcomes by leveraging routinely available radiology and pathology images. With further independent validation, this may afford a promising approach to improve risk stratification and better inform treatment strategies for cancer patients. Citation Format: Zhe Li, Yuming Jiang, Ruijiang Li. Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 2313.

Weakly Supervised AI for Efficient Analysis of 3D Pathology Samples

Analysis of 3D pathology samples using weakly supervised AI

Machine Learning for the Prediction of Prostate Cancer Biopsy Based on 3D Dynamic Contrast-Enhanced Ultrasound Quantification

Triage of 3D pathology data via 2.5D multiple-instance learning to guide pathologist assessments

Engineering the future of 3D pathology

Prostate cancer risk stratification via non-destructive 3D pathology with annotation-free gland segmentation and analysis

Nondestructive 3D pathology with analysis of nuclear features for prostate cancer risk assessment

Machine learning enabled prediction of digital biomarkers from whole slide histopathology images

3D Registration of pre-surgical prostate MRI and histopathology images via super-resolution volume reconstruction

Detecting Adverse Pathology of Prostate Cancer With a Deep Learning Approach Based on a 3D Swin-Transformer Model and Biparametric MRI: A Multicenter Retrospective Study

Digital Volumetric Biopsy Cores Improve Gleason Grading of Prostate Cancer Using Deep Learning

Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

3D-Morphomics, Morphological Features on CT scans for lung nodule malignancy diagnosis

Artificial Intelligence for Digital and Computational Pathology

Path-50. Ai-Powered Automated Tissue Segmentation Improves Outcome Stratification In Glioblastoma

Multiparametric Deep Learning Tissue Signatures for a Radiological Biomarker of Breast Cancer: Preliminary Results

3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding

Harnessing non-destructive 3D pathology

When Machine Vision Meets Histology: A Comparative Evaluation of Model Architecture for Classification of Histology Sections