Abstract:Significance: Hyperspectral imaging (HSI) is an emerging imaging modality for oncological applications and can improve cancer detection with digital pathology. Aim: The study aims to highlight the increased accuracy and sensitivity of detecting the margin of thyroid carcinoma in hematoxylin and eosin (H&E)-stained histological slides using HSI and data augmentation methods. Approach: Using an automated microscopic imaging system, we captured 2599 hyperspectral images from 65 H&E-stained human thyroid slides. Images were then preprocessed into 153,906 image patches of dimension 250 × 250 × 84 pixels . We modified the TimeSformer network architecture, which used alternating spectral attention and spatial attention layers. We implemented several data augmentation methods for HSI based on the RandAugment algorithm. We compared the performances of TimeSformer on HSI against the performances of pretrained ConvNext and pretrained vision transformers (ViT) networks on red, green, and blue (RGB) images. Finally, we applied attention unrolling techniques on the trained TimeSformer network to identify the biological features to which the network paid attention. Results: In the testing dataset, TimeSformer achieved an accuracy of 90.87%, a weighted F 1 score of 89.79%, a sensitivity of 91.50%, and an area under the receiving operator characteristic curve (AU-ROC) score of 97.04%. Additionally, TimeSformer produced thyroid carcinoma tumor margins with an average Jaccard score of 0.76 mm. Without data augmentation, TimeSformer achieved an accuracy of 88.23%, a weighted F 1 score of 86.46%, a sensitivity of 85.53%, and an AU-ROC score of 94.94%. In comparison, the ViT network achieved an 89.98% accuracy, an 88.14% weighted F 1 score, an 84.77% sensitivity, and a 96.17% AU-ROC. Our visualization results showed that the network paid attention to biological features. Conclusions: The TimeSformer model trained with hyperspectral histological data consistently outperformed conventional RGB-based models, highlighting the superiority of HSI in this context. Our proposed augmentation methods improved the accuracy, the F 1 score, and the sensitivity score.

A Dual Stream Network for Tumor Detection in Hyperspectral Images.

Tongue Tumor Detection in Hyperspectral Images Using Deep Learning Semantic Segmentation

Spectral-Spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification

Tongue Tumor Detection In Medical Hyperspectral Images

Automated Tumor Assessment Of Squamous Cell Carcinoma On Tongue Cancer Patients With Hyperspectral Imaging

Pixel-level Tumor Margin Assessment of Surgical Specimen with Hyperspectral Imaging and Deep Learning Classification.

Multi-spectral video endoscopy system for the detection of cancerous tissue

Joint Diagnostic Method of Tumor Tissue Based on Hyperspectral Spectral-Spatial Transfer Features

Hyperspectral Microscopic Imaging for Automatic Detection of Head and Neck Squamous Cell Carcinoma Using Histologic Image and Machine Learning.

Hyperspectral Imaging for Glioblastoma Surgery: Improving Tumor Identification Using a Deep Spectral-Spatial Approach

Detection and margin assessment of thyroid carcinoma with microscopic hyperspectral imaging using transformer networks

Explainable liver tumor delineation in surgical specimens using hyperspectral imaging and deep learning

Superpixel-based Spectral Classification for the Detection of Head and Neck Cancer with Hyperspectral Imaging

Robust Tumor Segmentation with Hyperspectral Imaging and Graph Neural Networks

In Vivo Cancer Detection in Animal Model Using Hyperspectral Image Classification with Wavelet Feature Extraction.

An Intraoperative Visualization System Using Hyperspectral Imaging To Aid In Brain Tumor Delineation

An Ensemble Learning Method for Detection of Head and Neck Squamous Cell Carcinoma Using Polarized Hyperspectral Microscopic Imaging

Deformable Registration of Histological Cancer Margins to Gross Hyperspectral Images using Demons

Blood Cancer Diagnosis Using Hyperspectral Imaging Assisted with Forward Searching Method and Machine Learning

Hyperspectral imaging for head and neck cancer detection: specular glare and variance of the tumor margin in surgical specimens

Intelligent tumor tissue classification for Hybrid Health Care Units