Abstract:Gastrointestinal endoscopic image analysis presents significant challenges, such as considerable variations in quality due to the challenging in-body imaging environment, the often-subtle nature of abnormalities with low interobserver agreement, and the need for real-time processing. These challenges pose strong requirements on the performance, generalization, robustness and complexity of deep learning-based techniques in such safety-critical applications. While Convolutional Neural Networks (CNNs) have been the go-to architecture for endoscopic image analysis, recent successes of the Transformer architecture in computer vision raise the possibility to update this conclusion. To this end, we evaluate and compare clinically relevant performance, generalization and robustness of state-of-the-art CNNs and Transformers for neoplasia detection in Barrett's esophagus. We have trained and validated several top-performing CNNs and Transformers on a total of 10,208 images (2,079 patients), and tested on a total of 7,118 images (998 patients) across multiple test sets, including a high-quality test set, two internal and two external generalization test sets, and a robustness test set. Furthermore, to expand the scope of the study, we have conducted the performance and robustness comparisons for colonic polyp segmentation (Kvasir-SEG) and angiodysplasia detection (Giana). The results obtained for featured models across a wide range of training set sizes demonstrate that Transformers achieve comparable performance as CNNs on various applications, show comparable or slightly improved generalization capabilities and offer equally strong resilience and robustness against common image corruptions and perturbations. These findings confirm the viability of the Transformer architecture, particularly suited to the dynamic nature of endoscopic video analysis, characterized by fluctuating image quality, appearance and equipment configurations in transition from hospital to hospital. The code is made publicly available at: https://github.com/BONS-AI-VCA-AMC/Endoscopy-CNNs-vs-Transformers.

On the calibration of neural networks for histological slide-level classification

From modern CNNs to vision transformers: Assessing the performance, robustness, and classification strategies of deep learning models in histopathology

A comparative study between vision transformers and CNNs in digital pathology

A Comparative Study of Explainability Methods for Whole Slide Classification of Lymph Node Metastases using Vision Transformers

ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise

Confidence Calibration of Classifiers with Many Classes

Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification

Understanding Calibration of Deep Neural Networks for Medical Image Classification

Transformers, convolutional neural networks, and few-shot learning for classification of histopathological images of oral cancer

Multi-Head Multi-Loss Model Calibration

Vision Transformer-Based Deep Learning for Histologic Classification of Endometrial Cancer

Will Transformers change gastrointestinal endoscopic image analysis? A comparative analysis between CNNs and Transformers, in terms of performance, robustness and generalization

Improving High Resolution Histology Image Classification with Deep Spatial Fusion Network

Revisiting the Calibration of Modern Neural Networks

Time to Embrace Natural Language Processing (NLP)-based Digital Pathology: Benchmarking NLP- and Convolutional Neural Network-based Deep Learning Pipelines

HistoTransfer: Understanding Transfer Learning for Histopathology

Data Alchemy: Mitigating Cross-Site Model Variability Through Test Time Data Calibration

Studying the Impact of Augmentations on Medical Confidence Calibration

Decoupling Feature Extraction and Classification Layers for Calibrated Neural Networks

Scalable deep learning artificial intelligence histopathology slide analysis and validation