Abstract:Background: Medical image classification is crucial for accurate and efficient diagnosis, and deep learning frameworks have shown significant potential in this area. When a general learning deep model is directly deployed to a new dataset with heterogeneous features, the effect of domain shifts is usually ignored, which degrades the performance of deep learning models and leads to inaccurate predictions. Purpose: This study aims to propose a framework that utilized the cross-modality domain adaptation and accurately diagnose and classify MRI scans and domain knowledge into stable and vulnerable plaque categories by a modified Vision Transformer (ViT) model for the classification of MRI scans and transformer model for domain knowledge classification. Methods: This study proposes a Hybrid Vision Inspired Transformer (HViT) framework that employs a convolutional layer for image pre-processing and normalization and a 3D convolutional layer to enable ViT to classify 3D images. Our proposed HViT framework introduces a slim design with a multi-branch network and channel attention, improving patch embedding extraction and information learning. Auxiliary losses target shallow features, linking them with deeper ones, enhancing information gain, and model generalization. Furthermore, replacing the MLP Head with RNN enables better backpropagation for improved performance. Moreover, we utilized a modified transformer model with LSTM positional encoding and Golve word vector to classify domain knowledge. By using ensemble learning techniques, specifically stacking ensemble learning with hard and soft prediction, we combine the predictive power of both models to address the cross-modality domain adaptation problem and improve overall performance. Results: The proposed framework achieved an accuracy of 94.32% for carotid artery plaque classification into stable and vulnerable plaque by addressing the cross-modality domain adaptation problem and improving overall performance. Conclusion: The model was further evaluated using an independent dataset acquired from different hardware protocols. The results demonstrate that the proposed deep learning model significantly improves the generalization ability across different MRI scans acquired from different hardware protocols without requiring additional calibration data.

Application of Vision-Series Transformer in Screening for Coronary Heart Diseases Using Coronary CT Angiography.

Vision Transformer for Efficient Chest X-ray and Gastrointestinal Image Classification

Using Vision Transformers in 3-D Medical Image Classifications

Research and implementation of multi-disease diagnosis on chest X-ray based on vision transformer

HViT: Hybrid vision inspired transformer for the assessment of carotid artery plaque by addressing the cross-modality domain adaptation problem in MRI

End to End Vision Transformer Architecture for Brain Stroke Assessment Based on Multi-Slice Classification and Localization Using Computed Tomography.

Vision Transformer-based recognition of diabetic retinopathy grade

Vision transformer introduces a new vitality to the classification of renal pathology

Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from CT-radiography

Heart Disease Detection using Vision-Based Transformer Models from ECG Images

Efficiently Training Vision Transformers on Structural MRI Scans for Alzheimer's Disease Detection

Transformer Network for Significant Stenosis Detection in CCTA of Coronary Arteries

Time-Distributed Vision Transformer Stacked With Transformer for Heart Failure Detection Based on Echocardiography Video

COVID-19 detection using ViT transformer-based approach from Computed Tomography Images

Vision transformer-convolution for breast cancer classification using mammography images: A comparative study

KDViT: COVID-19 diagnosis on CT-scans with knowledge distillation of vision transformer

Automated retinal disease classification using hybrid transformer model (SViT) using optical coherence tomography images

Leveraging a Vision Transformer Model to Improve Diagnostic Accuracy of Cardiac Amyloidosis With Cardiac Magnetic Resonance

COVID-VIT: Classification of COVID-19 from CT chest images based on vision transformer models

Improving diagnosis and prognosis of lung cancer using vision transformers: A scoping review

The auto segmentation for cardiac structures using a dual-input deep learning network based on vision saliency and transformer