Abstract:Checkpoint blockade immunotherapy is a cornerstone of lung cancer treatment, but there is a need to improve the identification of patients who will respond favorably. Here, we explored a deep learning approach to predict immunotherapy outcomes from hematoxylin and eosin (H&E) images in non-small cell lung cancer (NSCLC). We included 150 unique cases with metastatic NSCLC (113 adenocarcinoma, 29 squamous cell, 8 other) treated with anti-PD-1/PD-L1 immunotherapy (56 nivolumab, 49 atezolizumab, 44 pembrolizumab, 1 durvalumab) as mono or combination (14 with chemotherapy, 1 with ipilimumab) therapy in a single institution. Each case consisted of a representative H&E whole slide image (53 biopsies, 50 needle core biopsies, 47 resections) obtained prior to immunotherapy, and the outcome reported as the 1-year overall survival (OS). PD-L1 status (tumor proportion score ≥ 1%) was known for 70 cases. We preprocessed the H&E images using two deep learning models previously developed using The Cancer Genome Atlas dataset. First, we used a classification model to identify tumor regions and randomly sampled a fixed number of tumor patches for each case. Then, we used a self-supervised pathology foundation model to obtain a compressed visual representation of each patch, known as an embedding. Next, using our dataset, we trained a deep multiple instance learning (DeepMIL) model with a gated attention mechanism to predict the binary 1-year OS status (0=deceased, 1=alive) for each case. As a baseline, we also trained a linear-probe (logistic regression) model using the averaged embeddings. Given the small dataset size, 5-fold cross-validation was used to train and evaluate both the DeepMIL and linear-probe models, with cases randomly split across folds. For evaluation, we used survival analysis to compare the 0/1 case groups. Overall, across all 150 cases, univariable Cox regression showed that 1-year OS was more strongly associated with the DeepMIL status (46/104, HR=0.55, p=0.03) than the linear-probe status (55/95, HR=0.81, p=0.44). Results were consistent on the subset of 70 cases with known PD-L1 status, whereby OS was most strongly associated with the DeepMIL status (19/51, HR=0.40, p=0.04) compared to the linear-probe status (31/39, HR=0.46, p=0.09) and PD-L1 status (30/40, HR=0.65, p=0.32). In multivariable Cox regression adjusting for age group and smoking status, OS remained more strongly associated with the DeepMIL status (HR=0.45, p=0.08) than PD-L1 status (HR=0.72, p=0.47). In conclusion, the DeepMIL status predicted from H&E images showed a stronger association with outcomes compared to PD-L1 status, a standard biomarker for immunotherapy in NSCLC. These exploratory results demonstrate the potential of deep learning using pathology foundation models to improve immunotherapy outcomes prediction, even with small datasets. Such approaches may even enable the discovery of novel biomarkers from H&E images to advance precision medicine. Citation Format: Jessica Loo, Yang Wang, Pok Fai Wong, Ellery Wulczyn, Jeremy Lai, Peter Cimermancic, David F. Steiner, Shamira S. Weaver. Predicting immunotherapy outcomes from H&E images in lung cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular s); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl) nr 7380.

Abstract 6175: Deep learning-based survival prediction for hepatocellular carcinoma using whole slide images

Deep learning-based survival prediction for multiple cancer types using histopathology images

Abstract 2313: Multi-modal deep learning to predict cancer outcomes by integrating radiology and pathology images

Deep Learning-Based Prediction of Post-treatment Survival in Hepatocellular Carcinoma Patients Using Pre-treatment CT Images and Clinical Data

Abstract 7379: Deep learning based risk stratification of pre-operative CT scans is prognostic of overall survival in kidney cancers

Prediction of microvascular invasion and pathological differentiation of hepatocellular carcinoma based on a deep learning model

Interpretable Survival Prediction for Colorectal Cancer using Deep Learning

Clinically Applicable Deep Learning for Predicting Prognosis of Hepatocellular Carcinoma Directly From Histology Images

Abstract 7380: Predicting immunotherapy outcomes from H&E images in lung cancer

Probabilistic ratiocination of hepatocellular carcinoma after resection: evaluation of expected to be promising approaches

Outcome-Supervised Deep Learning on Pathologic Whole Slide Images for Survival Prediction of Immunotherapy in Patients with Non-Small Cell Lung Cancer

Deep learning classification of uveal melanoma based on histopathological images and identification of a novel indicator for prognosis of patients

A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images

Deep Learning Analysis of CT Images Reveals High-Grade Pathological Features to Predict Survival in Lung Adenocarcinoma

Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images

Deep Learning to Develop Transcriptomic Model for Survival Prediction in TCGA Patients with Hepatocellular Carcinoma.

Development of a Deep Learning Model to Assist with Diagnosis of Hepatocellular Carcinoma

Combining a deep learning model with clinical data better predicts hepatocellular carcinoma behavior following surgery

Abstract 2310: Integrating real-world histopathological and clinicogenomic data from 1799 lung cancer patients by applying unsupervised deep learning

Deep learning-derived spatial organization features on histopathology images to predict prognosis in patients with colorectal liver metastasis, after hepatectomy.

Deep Learning for Survival Analysis in Breast Cancer with Whole Slide Image Data