Abstract:Purpose Vision Transformers recently achieved a competitive performance compared with CNNs due to their excellent capability of learning global representation. However, there are two major challenges when applying them to 3D image segmentation: i) Because of the large size of 3D medical images, comprehensive global information is hard to capture due to the enormous computational costs. ii) Insufficient local inductive bias in Transformers affects the ability to segment detailed features such as ambiguous and subtly defined boundaries. Hence, to apply the Vision Transformer mechanism in the medical image segmentation field, the above challenges need to be overcome adequately. Methods We propose a hybrid paradigm, called Variable-Shape Mixed Transformer (VSmTrans), that integrates self-attention and convolution and can enjoy the benefits of free learning of both complex relationships from the self-attention mechanism and the local prior knowledge from convolution. Specifically, we designed a Variable-Shape self-attention mechanism, which can rapidly expand the receptive field without extra computing cost and achieve a good trade-off between global awareness and local details. In addition, the parallel convolution paradigm introduces strong local inductive bias to facilitate the ability to excavate details. Meanwhile, a pair of learnable parameters can automatically adjust the importance of the above two paradigms. Extensive experiments were conducted on two public medical image datasets with different modalities: the AMOS CT dataset and the BraTS2021 MRI dataset. Results Our method achieves the best average Dice scores of 88.3% and 89.7% on these datasets, which are superior to the previous state-of-the-art Swin Transformer-based and CNN-based architectures. A series of ablation experiments were also conducted to verify the efficiency of the proposed hybrid mechanism and the components and explore the effectiveness of those key parameters in VSmTrans. Conclusions The proposed hybrid Transformer-based backbone network for 3D medical image segmentation can tightly integrate self-attention and convolution to exploit the advantages of these two paradigms. The experimental results demonstrate our method's superiority compared to other state-of-the-art methods. The hybrid paradigm seems to be most appropriate to the medical image segmentation field. The ablation experiments also demonstrate that the proposed hybrid mechanism can effectively balance large receptive fields with local inductive biases, resulting in highly accurate segmentation results, especially in capturing details. Our code is available at https://github.com/qingze-bai/VSmTrans.

Improved Two-Stage Transfer Learning Approach for ViT-Based Myocardial Infarction Detection

A Fully Automated Two-Stage Segmentation Approach for Late Gadolinium-Enhanced Cardiac Magnetic Resonance Images in Personalized Cardiac Modeling

MMViT-Seg: A Lightweight Transformer and CNN Fusion Network for COVID-19 Segmentation.

A hybrid ResNet-ViT approach to bridge the global and local features for myocardial infarction detection

Enhancing Cardiac MRI Segmentation via Classifier-Guided Two-Stage Network and All-Slice Information Fusion Transformer

HViT: Hybrid vision inspired transformer for the assessment of carotid artery plaque by addressing the cross-modality domain adaptation problem in MRI

Two-Phase Segmentation Approach for Accurate Left Ventricle Segmentation in Cardiac MRI using Machine Learning

Automatic pipeline for segmentation of LV myocardium on quantitative MR T1 maps using deep learning model and computation of radial T1 and ECV values

NVTrans-UNet: Neighborhood vision transformer based U-Net for multi-modal cardiac MR image segmentation

The auto segmentation for cardiac structures using a dual-input deep learning network based on vision saliency and transformer

Ensemble learning of myocardial displacements for myocardial infarction detection in echocardiography

Detection of myocardial infarction based on novel deep transfer learning methods for urban healthcare in smart cities

Automated T1 and T2 mapping segmentation on cardiovascular magnetic resonance imaging using deep learning

End to End Vision Transformer Architecture for Brain Stroke Assessment Based on Multi-Slice Classification and Localization Using Computed Tomography.

VSmTrans: A Hybrid Paradigm Integrating Self-attention and Convolution for 3D Medical Image Segmentation

Automated segmentation of biventricular contours in tissue phase mapping using deep learning

MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation

Automatic Detection of Decreased Ejection Fraction and Left Ventricular Hypertrophy on 4D Cardiac CTA: Use of Artificial Intelligence with Transfer Learning to Facilitate Multi-Site Operations

Leveraging a Vision Transformer Model to Improve Diagnostic Accuracy of Cardiac Amyloidosis With Cardiac Magnetic Resonance

Exploiting Data-Efficient Image Transformer-Based Transfer Learning for Valvular Heart Diseases Detection