Abstract:e13593 Background: Precision oncology increasingly relies on accurate classification of the tumor microenvironment (TME), which influences therapeutic responses and prognostication in cancer. While histological observation via H&E staining remains the gold standard due to the detailed spatial context and morphological information provided, its lack of reproducibility limits its use in clinics. Next-generation sequencing (NGS) reliably provides valuable insights into the molecular landscape of tumors. Here we debut an unsupervised machine-learning approach that leverages both the rich detail in H&E slides and the reproducibility of NGS to enhance TME classification. Methods: An ML-based image analysis algorithm was developed using a convolutional neural network (CNN) deep learning model. Using over 600 images from the TCGA COAD and READ datasets and from 139 colorectal cancer (CRC) samples in an internal cohort, we trained the CNN model to curate features from small patches of whole slide images (WSIs). Then, clusters were created from these H&E WSIs without prior labeling. The clusters were validated by experienced pathologists to confirm cluster homogeneity and filter artifacts. We trained a classifier to categorize molecular features from the WSIs into TME subtypes. Performance was measured using the area under the receiver operating characteristic curve (AUC-ROC), based on the model's ability to correctly classify the Fibrotic (F) and Immune Desert (D) TME subtypes (1), as well as MSI status identified by whole exome sequencing. The F subtype corresponds with high stromal content, low immune infiltration, and the worst prognosis. The D subtype corresponds with a cold immune TME and an intermediate prognosis. Progression-free survival (PFS) data from TCGA were used to examine prognostic significance. Results: We obtained 0.86, 0.79, 0.72, and 0.71 AUC scores for predicting the ID and F TMEs, MSI status, and KRAS mutation status, respectively. PFS data analysis showed our H&E-based approach to yield similar or better prognostic significance compared to NGS-based TME classification (ID vs others, p=0.05 (H&E) and 0.2 (NGS); F vs others, p=0.07 (H&E) and 0.09 (NGS)). Our findings showed a notable correlation with RNA- and DNA-based predictions of ID and F TMEs and MSI, with enhanced precision and more robust subtype identification while achieving high embedding consistency. Conclusions: By harnessing the inherently rich detail in histological images, our H&E-based approach complements NGS-based TME analysis. Our findings advocate for the integration of advanced image analysis into the standard TME characterization workflow to refine prognostic and therapeutic strategies for CRC. We will expand this algorithm for use with other cancer types to realize its potential as a promising adjunct to NGS for precise TME classification. 1. Bagaev et al., 2021.

Weakly-supervised learning for image-based classification of primary melanomas into genomic immune subgroups

Immune subtyping of melanoma whole slide images using multiple instance learning

Weakly supervised deep learning image analysis can differentiate melanoma from naevi on haematoxylin and eosin‐stained histopathology slides

Automated identification of malignancy in whole-slide pathological images: identification of eyelid malignant melanoma in gigapixel pathological slides using deep learning

Machine-learning convergent melanocytic morphology despite noisy archival slides

Deep Learning-Based Accurate Diagnosis of Eyelid Malignant Melanoma from Gigapixel Pathologic Slides

Using Machine Learning Algorithms to Predict Immunotherapy Response in Patients with Advanced Melanoma

Molecular Classification of Breast Cancer Using Weakly Supervised Learning

Prediction of molecular subclasses of uveal melanoma by deep learning using routine haematoxylin-eosin-stained tissue slides

Prediction of Tumor-Associated Macrophages and Immunotherapy Benefits Using Weakly Supervised Contrastive Learning in Breast Cancer Pathology Images

Deep learning-based classification of breast cancer molecular subtypes from H&E whole-slide images

Whole slide image-based weakly supervised deep learning for predicting major pathological response in non-small cell lung cancer following neoadjuvant chemoimmunotherapy: a multicenter, retrospective, cohort study

Capturing global spatial context for accurate cell classification in skin cancer histology

Automated differentiation of skin melanocytes from keratinocytes in high‐resolution histopathology images using a weakly‐supervised deep‐learning framework

Using Whole Slide Image Representations from Self-Supervised Contrastive Learning for Melanoma Concordance Regression

Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology

Abstract 4951: Integrative multi-omic machine learning model predicts neoadjuvant immunotherapy response using molecular data and deep learning-derived features from digital pathology

Clinical-grade computational pathology using weakly supervised deep learning on whole slide images

An unsupervised H&E-based machine-learning approach for precise prediction of tumor microenvironment subtypes.

Deep learning classification of uveal melanoma based on histopathological images and identification of a novel indicator for prognosis of patients

Deep Learning-Based Classification of Early-Stage Mycosis Fungoides and Benign Inflammatory Dermatoses on H&E-Stained Whole-Slide Images: A Retrospective, Proof-of-Concept Study