Abstract:Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the "clustering-constrained-attention multiple-instance learning" (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1) nr B082.

Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype

Deep learning-based classification of breast cancer molecular subtypes from H&E whole-slide images

hist2RNA: An efficient deep learning architecture to predict gene expression from breast cancer histopathology images

Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images

Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images

Deep learning models for histologic grading of breast cancer and association with disease prognosis

Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review

Predicting Breast Cancer Gene Expression Signature by Applying Deep Convolutional Neural Networks From Unannotated Pathological Images

Artificial Intelligence Predicts Multiclass Molecular Signatures and Subtypes Directly from Breast Cancer Histology

Data-Efficient Computational Pathology Platform for Faster and Cheaper Breast Cancer Subtype Identifications: Development of a Deep Learning Model

Abstract B082: Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning

Deep Learning for identifying radiogenomic associations in breast cancer

Image analysis-based identification of high risk ER-positive, HER2-negative breast cancers

Abstract 5354: Prediction of OncotypeDX high risk group for chemotherapy benefit in breast cancer by deep learning analysis of hematoxylin and eosin-stained whole slide images

Predicting Ki67, ER, PR, and HER2 Statuses from H&E-stained Breast Cancer Images

Determining breast cancer biomarker status and associated morphological features using deep learning

An unsupervised H&E-based machine-learning approach for precise prediction of tumor microenvironment subtypes.

Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains

Predicting early breast cancer recurrence from histopathological images in the Carolina Breast Cancer Study

Abstract P5-02-33: Deep learning-based assessment of HER2-low expression on breast cancer H&E digital whole slide images

Prediction of BRCA Gene Mutation in Breast Cancer Based on Deep Learning and Histopathology Images