Abstract:Traditional Chinese medicine (TCM) observation diagnosis images (including facial and tongue images) provide essential human body information, holding significant importance in clinical medicine for diagnosis and treatment . TCM prescriptions, known for their simplicity, non-invasiveness, and low side effects, have been widely applied worldwide. Exploring automated herbal prescription construction based on visual diagnosis holds vital value in delving into the correlation between external features and herbal prescriptions and offering medical services in mobile healthcare systems. To effectively integrate multi-perspective visual diagnosis images and automate prescription construction, this study proposes a multi-herb recommendation framework based on Visual Transformer and multi-label classification. The framework comprises three key components: image encoder, label embedding module, and cross-modal fusion classification module. The image encoder employs a dual-stream Visual Transformer to learn dependencies between different regions of input images, capturing both local and global features. The label embedding module utilizes Graph Convolutional Networks to capture associations between diverse herbal labels. Finally, two Multi-Modal Factorized Bilinear modules are introduced as effective components to fuse cross-modal vectors, creating an end-to-end multi-label image-herb recommendation model. Through experimentation with real facial and tongue images and generating prescription data closely resembling real samples. The precision is 50.06 %, the recall rate is 48.33 %, and the F1-score is 49.18 %. This study validates the feasibility of automated herbal prescription construction from the perspective of visual diagnosis. Simultaneously, it provides valuable insights for constructing herbal prescriptions automatically from more physical information.

Spectrum-image Dual-Modality Fusion Empowered Accurate and Efficient Classification System for Traditional Chinese Medicine

Cross-Domain Learning Based Traditional Chinese Medicine Medical Record Classification.

A Hyperspectral Imaging Approach for Classifying Geographical Origins of Rhizoma Atractylodis Macrocephalae Using the Fusion of Spectrum-Image in VNIR and SWIR Ranges (Vnir-Swir-fusi)

Data Fusion of Laser Induced Breakdown Spectroscopy (LIBS) and Infrared Spectroscopy (IR) Coupled with Random Forest (RF) for the Classification and Discrimination of Compound Salvia Miltiorrhiza

Topology and Semantic Information Fusion Classification Network Based on Hyperspectral Images of Chinese Herbs

Feature‐level Fusion of Laser‐induced Breakdown Spectroscopy and Raman Spectroscopy for Improving Support Vector Machine in Clinical Bacteria Identification

Rapid identification of Radix Astragali by data fusion of laser-induced breakdown spectroscopy and Raman spectroscopy coupled with deep learning

An End-to-End Traditional Chinese Medicine Constitution Assessment System Based on Multimodal Clinical Feature Representation and Fusion

Quality assessment of traditional Chinese medicine based on data fusion combined with machine learning: A review

A multimodal deep architecture for traditional Chinese medicine diagnosis

Image recognition of traditional Chinese medicine based on deep learning

Development status of novel spectral imaging techniques and application to traditional Chinese medicine

Hyperspectral Image Classification for Foreign Bodies in Medicine Based on Spatial-Spectral Fusion Feature

TCMFVis:A Visual Analytics System Toward Bridging Together Traditional Chinese Medicine and Modern Medicine

A Vision Technology Fusion Framework for Sorting Panax notoginseng

A Robust TabNet-Based Multi-Classification Algorithm for Infrared Spectral Data of Chinese Herbal Medicine with High-Dimensional Small Samples

Two-stream vision transformer based multi-label recognition for TCM prescriptions construction

Optimization of Machine Learning Classification Models for Tumor Cells Based on Cell Elements Heterogeneity with Laser-Induced Breakdown Spectroscopy.

Multi-modal Fusion for Diabetes Mellitus and Impaired Glucose Regulation Detection

Application of Data Fusion in Traditional Chinese Medicine: A Review

Classification method of traditional Chinese medicine compound decoction duration based on multi-dimensional feature weighted fusion