Abstract:Importance: Immunohistochemistry (IHC) is the most widely used assay for identification of molecular biomarkers. However, IHC is time consuming and costly, depends on tissue-handling protocols, and relies on pathologists' subjective interpretation. Image analysis by machine learning is gaining ground for various applications in pathology but has not been proposed to replace chemical-based assays for molecular detection. Objective: To assess the prediction feasibility of molecular expression of biomarkers in cancer tissues, relying only on tissue architecture as seen in digitized hematoxylin-eosin (H&E)-stained specimens. Design, setting, and participants: This single-institution retrospective diagnostic study assessed the breast cancer tissue microarrays library of patients from Vancouver General Hospital, British Columbia, Canada. The study and analysis were conducted from July 1, 2015, through July 1, 2018. A machine learning method, termed morphological-based molecular profiling (MBMP), was developed. Logistic regression was used to explore correlations between histomorphology and biomarker expression, and a deep convolutional neural network was used to predict the biomarker expression in examined tissues. Main outcomes and measures: Positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristics curve measures of MBMP for assessment of molecular biomarkers. Results: The database consisted of 20 600 digitized, publicly available H&E-stained sections of 5356 patients with breast cancer from 2 cohorts. The median age at diagnosis was 61 years for cohort 1 (412 patients) and 62 years for cohort 2 (4944 patients), and the median follow-up was 12.0 years and 12.4 years, respectively. Tissue histomorphology was significantly correlated with the molecular expression of all 19 biomarkers assayed, including estrogen receptor (ER), progesterone receptor (PR), and ERBB2 (formerly HER2). Expression of ER was predicted for 105 of 207 validation patients in cohort 1 (50.7%) and 1059 of 2046 validation patients in cohort 2 (51.8%), with PPVs of 97% and 98%, respectively, NPVs of 68% and 76%, respectively, and accuracy of 91% and 92%, respectively, which were noninferior to traditional IHC (PPV, 91%-98%; NPV, 51%-78%; and accuracy, 81%-90%). Diagnostic accuracy improved given more data. Morphological analysis of patients with ER-negative/PR-positive status by IHC revealed resemblance to patients with ER-positive status (Bhattacharyya distance, 0.03) and not those with ER-negative/PR-negative status (Bhattacharyya distance, 0.25). This suggests a false-negative IHC finding and warrants antihormonal therapy for these patients. Conclusions and relevance: For at least half of the patients in this study, MBMP appeared to predict biomarker expression with noninferiority to IHC. Results suggest that prediction accuracy is likely to improve as data used for training expand. Morphological-based molecular profiling could be used as a general approach for mass-scale molecular profiling based on digitized H&E-stained images, allowing quick, accurate, and inexpensive methods for simultaneous profiling of multiple biomarkers in cancer tissues.

Automated recognition of cell phenotypes in histology images based on membrane- and nuclei-targeting biomarkers

Artificial Intelligence Algorithms to Assess Hormonal Status From Tissue Microarrays in Patients With Breast Cancer

Automated Region-based Prostate Cancer Cell Nuclei Localization. Part of a Prognostic Modality Tool for Prostate Cancer Patients

Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network

A high-content image-based method for quantitatively studying context-dependent cell population dynamics

Automatic segmentation and supervised learning-based selection of nuclei in cancer tissue images

A color‐based tumor segmentation method for clinical ex vivo breast tissue assessment utilizing a multi‐contrast brightfield imaging strategy

Automated Analysis and Classification of Histological Tissue Features by Multi-Dimensional Microscopic Molecular Profiling

Molecular Predictors of 3D Morphogenesis by Breast Cancer Cell Lines in 3D Culture

Nondestructive, quantitative viability analysis of 3D tissue cultures using machine learning image segmentation

Learning deep features for dead and living breast cancer cell classification without staining

Automated quantification of aligned collagen for human breast carcinoma prognosis

Predicting Ki67, ER, PR, and HER2 Statuses from H&E-stained Breast Cancer Images

Multiplexed Barcoding Image Analysis for Immunoprofiling and Spatial Mapping Characterization in the Single-Cell Analysis of Paraffin Tissue Samples

Principal component-based image segmentation: a new approach to outline in vitro cell colonies

Automated segmentation of tissue images for computerized IHC analysis

Integrated tumor identification and automated scoring minimizes pathologist involvement and provides new insights to key biomarkers in breast cancer

Data-Efficient Computational Pathology Platform for Faster and Cheaper Breast Cancer Subtype Identifications: Development of a Deep Learning Model

High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers

Cell2Cell: Explorative Cell Interaction Analysis in Multi-Volumetric Tissue Data

Analysis of organoid and immune cell co-cultures by machine learning-empowered image cytometry