Abstract:Breast cancer prognostication guides treatment decisions, with transcriptomic assays like Prosigna and Oncotype DX providing valuable risk of recurrence (ROR) scores in ER+/HER2– patients. However, these tests are costly and require sufficient biopsy tissue for accuracy. Pathology images are routinely available and could expedite rapid risk stratification at lower cost. We propose a multi-modal, multi-task deep neural network that learns transcriptomic ROR using digital pathology images and hormone receptor expression status. Because the incorporation of both pathology images and clinical data has proven beneficial in other deep learning models, we hypothesized that incorporation of this complementary information would improve ROR predictions compared to using pathology images alone. We modified the "clustering-constrained-attention multiple-instance learning" (CLAM) deep learning method to perform multi-task regression and infer continuous risk scores (instead of discrete risk categories). We evaluated (1) PAM50 ROR-P, calculated using a multivariate model based on intrinsic subtype centroids and a proliferation score, and (2) a 21-gene assay recurrence score (RS) (research version of Oncotype DX), calculated from component scores relating to proliferation, ER, HER2, and invasion. Using data from 899 patients in the Carolina Breast Cancer Study, a population-based study of diverse patients including 50% Black women and 50% women under age 50, we constructed training (n=714), validation (n=93), and test (n=92) sets. Genomic data were assayed by Nanostring, and digital pathology was based on H&E-stained whole slide images (WSIs). Models were assessed using Pearson correlation between measured and predicted ROR-P or RS. The model with the highest correlations on the test set used H&E WSIs and ER/PR/HER2 expression status as inputs and generated 10 independent outputs: ROR-P, four PAM50 centroid correlations (i.e., Basal, HER2-enriched, Luminal A, and Luminal B), RS, and four RS component scores. Prediction was strong when considering all participants in the test set (ROR-P, 0.78; RS, 0.83, N, 92) but was reduced among more clinically homogeneous ER+/HER2– patients (ROR-P, 0.55; RS, 0.49; N, 49). For RS, correlations were higher in models that included ER/PR/HER2 (Overall, 0.83; ER+/HER2–, 0.50) than in those that relied on pathology alone (Overall, 0.60, ER+/HER2–, 0.34). Future models will consider strategies to enhance performance in ER+/HER2– patients, such as oversampling of ER+/HER2– patients and use of loss functions designed to focus learning on mis-predicted examples. In summary, multi-task, tissue-based regression deep learning models recapitulate transcription-based risk assays with high correlations and, with optimization (especially integration of additional clinicopathologic data), hold promise for personalized treatment decisions from early, routinely-collected biopsy images. Citation Format: Jakub R Kaczmarzyk, Luke A Torre-Healy, Richard A Moffitt, Rajarsi Gupta, Alina M Hamilton, Tahsin M Kurc, Katherine A Hoadley, Melissa A Troester, Joel H Saltz. Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Advances in Breast Cancer Research; 2023 Oct 19-22; San Diego, California. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_1) nr B082.

010 Enhanced outcome prediction in cutaneous squamous cell carcinoma using deep-learning and computational histopathology (cSCCnet)

Artificial Classification of Cervical Squamous Lesions in ThinPrep Cytologic Tests Using a Deep Convolutional Neural Network.

Evaluation of a Deep Learning Model for Metastatic Squamous Cell Carcinoma Prediction From Whole Slide Images

Artificial Intelligence-Empowered Cellular Morphometric Risk Score Improves Prognostic Stratification of Cutaneous Squamous Cell Carcinoma.

A 18F-FDG PET/CT-based Deep Learning-Radiomics-clinical Model for Prediction of Cervical Lymph Node Metastasis in Esophageal Squamous Cell Carcinoma

Deep learning-based diagnosis and survival prediction of patients with renal cell carcinoma from primary whole slide images

Predicting Lymph Node Metastasis from Primary Cervical Squamous Cell Carcinoma Based on Deep Learning in Histopathologic Images.

Explainable, federated deep learning model predicts disease progression risk of cutaneous squamous cell carcinoma

Cervical Cancer Metastasis and Recurrence Risk Prediction Based on Deep Convolutional Neural Network

Transcriptomic analysis of cutaneous squamous cell carcinoma reveals a multi-gene prognostic signature associated with metastasis

Predicting Metastasis Risk in Pancreatic Neuroendocrine Tumors Using Deep Learning Image Analysis

Multiresolution Application of Artificial Intelligence in Digital Pathology for Prediction of Positive Lymph Nodes From Primary Tumors in Bladder Cancer.

Abstract 5354: Prediction of OncotypeDX high risk group for chemotherapy benefit in breast cancer by deep learning analysis of hematoxylin and eosin-stained whole slide images

Abstract B082: Early risk stratification of ER+/HER2– breast cancer patients using digital pathology and multi-task, weakly-supervised deep learning

Development and validation of a deep learning model for improving detection of nonmelanoma skin cancers treated with Mohs micrographic surgery

Deep Learning–Based Classification of Epithelial–Mesenchymal Transition for Predicting Response to Therapy in Clear Cell Renal Cell Carcinoma

Automated cutaneous squamous cell carcinoma grading using deep learning with transfer learning

Abstract 1939: Multimodal prediction of metastatic relapse using federated deep learning outperforms state-of-the-art methods in soft-tissue sarcoma

Predicting cancer content in tiles of lung squamous cell carcinoma tumours with validation against pathologist labels

Cutaneous squamous cell carcinoma characterized by MALDI mass spectrometry imaging in combination with machine learning

Comparison of whole slide image–based deep learning algorithms and genomic classifiers for assessing the risk of prostate cancer metastasis in surgically treated patients.