Abstract:Background: Currently, the diagnosis of gastrointestinal diseases relies heavily on electronic endoscopy, which is not suitable for all individuals and insufficient for dynamic patient monitoring. Therefore, there is an urgent need for a convenient and noninvasive examination method to enhance the diagnosis and monitoring of gastrointestinal diseases. Methods: Since the tongue is a significant reflection of the digestive system and closely associated with gastrointestinal diseases, our study aims to develop a diagnostic framework based on tongue features. We have collected a dataset of 2,167 tongue coating images from 949 consecutive patients, including 905 images from healthy individuals and 1,262 images from patients with various gastrointestinal conditions. Notably, each patient sample may represent distinct gastrointestinal diseases. This study introduces a novel approach to information fusion detection, integrating hand-crafted and auto-encoded features, alongside disease detection employing Squeeze and Excitation (SE) and Slot attention mechanisms. These two branches respectively analyze image feature and spatial position information when utilizing tongue images for diagnosing gastrointestinal diseases. The amalgamation of predictions from both aspects yields the final detection outcome. Results: For both healthy and diseased cohorts, the optimal classification metrics were achieved: AUC = 0.886, ACC = 0.849, and TPR = 0.965, indicating satisfactory performance. Notably, our framework demonstrated promising results in detecting five common gastrointestinal diseases: Helicobacter pylori infection, bile reflux, reflux esophagitis, gastric erosion, and duodenal erosion. Ablation experiments underscored the efficacy of mixed features in multi-color space, showcasing an increase in AUC and ACC by 13.48 % and 12.05 % respectively. Furthermore, the attention mechanism, by focusing on the middle region of tongue images, contributed to a 6.12 % increase in AUC and a 6.35 % increase in ACC. Conclusions: In this study, we successfully develop and validate a diagnosis framework for gastrointestinal diseases based on tongue image features. By utilizing tongue images, we achieve intelligent diagnosis of gastrointestinal diseases, establishing tongue diagnosis as a convenient, cost-effective, and noninvasive method for diagnosing and screening gastrointestinal diseases.

Tongue feature dataset construction and real-time detection

Swin-ResNet: Research and Implementation of a Tooth-Marked Tongue Classification Method Combining ResNet-50 and Swin Transformer

A Deep Tongue Image Features Analysis Model for Medical Application.

Automatic Tongue Image Segmentation for Real-Time Remote Diagnosis

Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application.

An Automatic Recognition of Tooth- Marked Tongue Based on Tongue Region Detection and Tongue Landmark Detection Via Deep Learning.

Human-computer interaction based health diagnostics using ResNet34 for tongue image classification

Automatic Tongue Crack Extraction For Real-Time Diagnosis

Intelligent quality control of traditional chinese medical tongue diagnosis images based on deep learning

A novel tongue feature extraction method on mobile devices

Artificial intelligence in tongue diagnosis: Using deep convolutional neural network for recognizing unhealthy tongue with tooth-mark

Tongue feature recognition to monitor rehabilitation: deep neural network with visual attention mechanism

TISNet-Enhanced Fully Convolutional Network with Encoder-Decoder Structure for Tongue Image Segmentation in Traditional Chinese Medicine

A novel hyperspectral medical sensor for tongue diagnosis

Automatic Classification Framework of Tongue Feature Based on Convolutional Neural Networks

Tongue Images Classification Based on Constrained High Dispersal Network

Intelligent Tongue Diagnosis Model for Gastrointestinal Diseases Based on Tongue Images

Deep Learning Based Tongue Prickles Detection in Traditional Chinese Medicine

Deep Learning Multi-label Tongue Image Analysis and Its Application in a Population Undergoing Routine Medical Checkup

An Intelligent Tongue Diagnosis System via Deep Learning on the Android Platform

A Novel Automatic Tongue Coating Extraction Method In Tongue Diagnose Of Traditional Chinese Medicine