Abstract:Importance: Deep learning-based methods, such as the sliding window approach for cropped-image classification and heuristic aggregation for whole-slide inference, for analyzing histological patterns in high-resolution microscopy images have shown promising results. These approaches, however, require a laborious annotation process and are fragmented.Objective: To evaluate a novel deep learning method that uses tissue-level annotations for high-resolution histological image analysis for Barrett esophagus (BE) and esophageal adenocarcinoma detection.Design, Setting, and Participants: This diagnostic study collected deidentified high-resolution histological images (N = 379) for training a new model composed of a convolutional neural network and a grid-based attention network. Histological images of patients who underwent endoscopic esophagus and gastroesophageal junction mucosal biopsy between January 1, 2016, and December 31, 2018, at Dartmouth-Hitchcock Medical Center (Lebanon, New Hampshire) were collected.Main Outcomes and Measures: The model was evaluated on an independent testing set of 123 histological images with 4 classes: normal, BE-no-dysplasia, BE-with-dysplasia, and adenocarcinoma. Performance of this model was measured and compared with that of the current state-of-the-art sliding window approach using the following standard machine learning metrics: accuracy, recall, precision, and F1 score.Results: Of the independent testing set of 123 histological images, 30 (24.4%) were in the BE-no-dysplasia class, 14 (11.4%) in the BE-with-dysplasia class, 21 (17.1%) in the adenocarcinoma class, and 58 (47.2%) in the normal class. Classification accuracies of the proposed model were 0.85 (95% CI, 0.81-0.90) for the BE-no-dysplasia class, 0.89 (95% CI, 0.84-0.92) for the BE-with-dysplasia class, and 0.88 (95% CI, 0.84-0.92) for the adenocarcinoma class. The proposed model achieved a mean accuracy of 0.83 (95% CI, 0.80-0.86) and marginally outperformed the sliding window approach on the same testing set. The F1 scores of the attention-based model were at least 8% higher for each class compared with the sliding window approach: 0.68 (95% CI, 0.61-0.75) vs 0.61 (95% CI, 0.53-0.68) for the normal class, 0.72 (95% CI, 0.63-0.80) vs 0.58 (95% CI, 0.45-0.69) for the BE-no-dysplasia class, 0.30 (95% CI, 0.11-0.48) vs 0.22 (95% CI, 0.11-0.33) for the BE-with-dysplasia class, and 0.67 (95% CI, 0.54-0.77) vs 0.58 (95% CI, 0.44-0.70) for the adenocarcinoma class. However, this outperformance was not statistically significant.Conclusions and Relevance: Results of this study suggest that the proposed attention-based deep neural network framework for BE and esophageal adenocarcinoma detection is important because it is based solely on tissue-level annotations, unlike existing methods that are based on regions of interest. This new model is expected to open avenues for applying deep learning to digital pathology.

Predicting the Visual Attention of Pathologists Evaluating Whole Slide Images of Cancer

Visual attention analysis of pathologists examining whole slide images of Prostate cancer

Decoding the visual attention of pathologists to reveal their level of expertise

Object-based feedback attention in convolutional neural networks improves tumour detection in digital pathology

Dual Attention Model with Reinforcement Learning for Classification of Histology Whole-Slide Images

Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images

An attention-based multi-resolution model for prostate whole slide imageclassification and localization

A Joint Spatial and Magnification Based Attention Framework for Large Scale Histopathology Classification

Spatial Attention-Based Deep Learning System for Breast Cancer Pathological Complete Response Prediction with Serial Histopathology Images in Multiple Stains

Contrastive learning-based histopathological features infer molecular subtypes and clinical outcomes of breast cancer from unannotated whole slide images

Semantics-Aware Attention Guidance for Diagnosing Whole Slide Images

PATHS: A Hierarchical Transformer for Efficient Whole Slide Image Analysis

A Multi-scale Graph Network with Multi-head Attention for Histopathology Image Diagnosisn

Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides

Deep learning infers clinically relevant protein levels and drug response in breast cancer from unannotated pathology images

Robust ROI Detection in Whole Slide Images Guided by Pathologists' Viewing Patterns

Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks

Interpretable Classification of Pathology Whole-Slide Images Using Attention Based Context-Aware Graph Convolutional Neural Network

Self-supervised attention-based deep learning for pan-cancer mutation prediction from histopathology

Masked Attention as a Mechanism for Improving Interpretability of Vision Transformers