Abstract:Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the "clustering-constrained-attention multiple-instance learning" (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1) nr B082.

Bi-level Graph Learning Unveils Prognosis-Relevant Tumor Microenvironment Patterns in Breast Multiplexed Digital Pathology

Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens

Spatially Aware Graph Neural Networks and Cross-Level Molecular Profile Prediction in Colon Cancer Histopathology: a Retrospective Multi-Cohort Study.

Single-Cell Spatial Analysis of Histopathology Images for Survival Prediction Via Graph Attention Network

Deep learning-based tumor microenvironment segmentation is predictive of tumor mutations and patient survival in non-small-cell lung cancer

Deep-learning and transfer learning identify new breast cancer survival subtypes from single-cell imaging data

Predicting Gene Spatial Expression and Cancer Prognosis: An Integrated Graph and Image Deep Learning Approach Based on HE Slides

Biology-guided deep learning predicts prognosis and cancer immunotherapy response

An integrated network representation of multiple cancer-specific data for graph-based machine learning

Abstract 6191: Simultaneous prediction of tumor microenvironment biomarkers from pathology slides using multi-task deep regression

AI-Powered Interpretable Imaging Phenotypes Noninvasively Characterize Tumor Microenvironment Associated with Diverse Molecular Signatures and Survival in Breast Cancer

Deep Learning Framework for Comprehensive Molecular and Prognostic Stratifications of Triple-Negative Breast Cancer

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Abstract B082: Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning

Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning

Deep Learning and Transfer Learning Identify Breast Cancer Survival Subtypes from Single-Cell Imaging Data.

Tumor Microenvironment-Related Multigene Prognostic Prediction Model for Breast Cancer

Deep Learning-based Prediction of Breast Cancer Tumor and Immune Phenotypes from Histopathology

Deciphering breast cancer prognosis: a novel machine learning-driven model for vascular mimicry signature prediction

Exploration of a noninvasive radiomics classifier for breast cancer tumor microenvironment categorization and prognostic outcome prediction

A Disease Network-Based Deep Learning Approach for Characterizing Melanoma.