Abstract:e13593 Background: Precision oncology increasingly relies on accurate classification of the tumor microenvironment (TME), which influences therapeutic responses and prognostication in cancer. While histological observation via H&E staining remains the gold standard due to the detailed spatial context and morphological information provided, its lack of reproducibility limits its use in clinics. Next-generation sequencing (NGS) reliably provides valuable insights into the molecular landscape of tumors. Here we debut an unsupervised machine-learning approach that leverages both the rich detail in H&E slides and the reproducibility of NGS to enhance TME classification. Methods: An ML-based image analysis algorithm was developed using a convolutional neural network (CNN) deep learning model. Using over 600 images from the TCGA COAD and READ datasets and from 139 colorectal cancer (CRC) samples in an internal cohort, we trained the CNN model to curate features from small patches of whole slide images (WSIs). Then, clusters were created from these H&E WSIs without prior labeling. The clusters were validated by experienced pathologists to confirm cluster homogeneity and filter artifacts. We trained a classifier to categorize molecular features from the WSIs into TME subtypes. Performance was measured using the area under the receiver operating characteristic curve (AUC-ROC), based on the model's ability to correctly classify the Fibrotic (F) and Immune Desert (D) TME subtypes (1), as well as MSI status identified by whole exome sequencing. The F subtype corresponds with high stromal content, low immune infiltration, and the worst prognosis. The D subtype corresponds with a cold immune TME and an intermediate prognosis. Progression-free survival (PFS) data from TCGA were used to examine prognostic significance. Results: We obtained 0.86, 0.79, 0.72, and 0.71 AUC scores for predicting the ID and F TMEs, MSI status, and KRAS mutation status, respectively. PFS data analysis showed our H&E-based approach to yield similar or better prognostic significance compared to NGS-based TME classification (ID vs others, p=0.05 (H&E) and 0.2 (NGS); F vs others, p=0.07 (H&E) and 0.09 (NGS)). Our findings showed a notable correlation with RNA- and DNA-based predictions of ID and F TMEs and MSI, with enhanced precision and more robust subtype identification while achieving high embedding consistency. Conclusions: By harnessing the inherently rich detail in histological images, our H&E-based approach complements NGS-based TME analysis. Our findings advocate for the integration of advanced image analysis into the standard TME characterization workflow to refine prognostic and therapeutic strategies for CRC. We will expand this algorithm for use with other cancer types to realize its potential as a promising adjunct to NGS for precise TME classification. 1. Bagaev et al., 2021.

Deep Representation Learning of Tissue Metabolome and Computed Tomography Images Annotates Non-invasive Classification and Prognosis Prediction of NSCLC

Deep representation learning of tissue metabolome and computed tomography annotates NSCLC classification and prognosis

Predicting the Tumor Microenvironment Composition and Immunotherapy Response in Non-Small Cell Lung Cancer from Digital Histopathology Images

Exploring non-invasive precision treatment in non-small cell lung cancer patients through deep learning radiomics across imaging features and molecular phenotypes

Multi-Size Deep Learning Based Preoperative Computed Tomography Signature for Prognosis Prediction of Colorectal Cancer

Deep Multi-Modality Collaborative Learning for Distant Metastases Predication in PET-CT Soft-Tissue Sarcoma Studies.

Exploration of a noninvasive radiomics classifier for breast cancer tumor microenvironment categorization and prognostic outcome prediction

Automated Cancer Diagnostics via Analysis of Optical and Chemical Images by Deep and Shallow Learning

Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging

Deep learning classification of lung cancer histology using CT images

An unsupervised H&E-based machine-learning approach for precise prediction of tumor microenvironment subtypes.

A transformer-based deep learning approach for classifying brain metastases into primary organ sites using clinical whole brain MRI

A Deep Learning Approach for Tissue Spatial Quantification and Genomic Correlations of Histopathological Images

Multimodal radiomics-based methods using deep learning for prediction of brain metastasis in non-small cell lung cancer with 18F-FDG PET/CT images

Towards radiologist-level cancer risk assessment in CT lung screening using deep learning

Multiparametric Deep Learning Tissue Signatures for a Radiological Biomarker of Breast Cancer: Preliminary Results

Harnessing TME depicted by histological images to improve cancer prognosis through a deep learning system

Deep Learning Based Time-to-event Analysis with PET, CT and Joint PET/CT for Head and Neck Cancer Prognosis

Predicting benefit from immune checkpoint inhibitors in patients with non-small-cell lung cancer by CT-based ensemble deep learning: a retrospective study

Non-invasive tumor microenvironment evaluation and treatment response prediction in gastric cancer using deep learning radiomics

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images