Abstract:Current predictive biomarkers generally leverage technologies such as immunohis-tochemistry or genetic analysis, which may require specialized equipment, be time-intensive to deploy, or incur human error. In this paper, we present an alternative approach for the development and deployment of a class of predictive biomarkers, leveraging deep learning on digital images of hematoxylin and eosin (H&E)-stained biopsy samples to simultaneously predict a range of molecular factors that are relevant to treatment selection and response. Our framework begins with the training of a pan-solid tumor H&E foundation model, which can generate a universal featurization of H&E-stained tissue images. This featurization becomes the input to machine learning models that perform multi-target, pan-cancer imputation. For a set of 352 drug targets, we show the ability to predict with high accuracy: copy number amplifications, target RNA expression, and an RNA-derived “amplification signature” that captures the transcriptional consequences of an amplification event. We facilitate exploratory analyses by making broad predictions initially. Having identified the subset of biomarkers relevant to a patient population of interest, we develop specialized machine learning models, built on the same foundational featurization, which achieve even higher performance for key biomarkers in tumor types of interest. Moreover, our models are robust, generalizing with minimal loss of performance across different patient populations. By generating imputations from tile-level featurizations, we enable spatial overlays of molecular annotations on top of whole-slide images. These annotation maps provide a clear means of interpreting the histological correlates of our model’s predictions, and align with features identified by expert pathologist review. Overall, our work demonstrates a flexible and scalable framework for imputing molecular measurements from H&E, providing a generalizable approach to the development and deployment of predictive biomarkers for targeted therapeutics in cancer.

Machine learning in computational histopathology: Challenges and opportunities

Deep Learning Models for Digital Pathology

State of machine and deep learning in histopathological applications in digestive diseases

Deep learning in histopathology: the path to the clinic

Deep learning in cancer genomics and histopathology

The Role of Machine Learning in Cardiovascular Pathology

A review of machine learning approaches, challenges and prospects for computational tumor pathology

Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Review

Deep neural network models for computational histopathology: A survey

Machine learning enabled prediction of digital biomarkers from whole slide histopathology images

Artificial Intelligence for Digital and Computational Pathology

Objective Diagnosis for Histopathological Images Based on Machine Learning Techniques: Classical Approaches and New Trends

Deep Learning-Based Histopathological Image Analysis for Automated Detection and Staging of Melanoma

Histopathological Images Analysis and Predictive Modeling Implemented in Digital Pathology—Current Affairs and Perspectives

Methods for Nuclei Detection, Segmentation, and Classification in Digital Histopathology: A Review—Current Status and Future Potential

Colorectal Cancer Outcome Prediction from H&E Whole Slide Images using Machine Learning and Automatically Inferred Phenotype Profiles

Computational pathology in cancer diagnosis, prognosis, and prediction – present day and prospects

The Emergence of Pathomics

Applying Machine Learning for Integration of Multi-Modal Genomics Data and Imaging Data to Quantify Heterogeneity in Tumour Tissues

Review of deep learning-based pathological image classification: From task-specific models to foundation models