Abstract:Background and Objective: Transformer, which is notable for its ability of global context modeling, has been used to remedy the shortcomings of Convolutional neural networks (CNN) and break its dominance in medical image segmentation. However, the self-attention module is both memory and computational inefficient, so many methods have to build their Transformer branch upon largely downsampled feature maps or adopt the tokenized image patches to fit their model into accessible GPUs. This patch-wise operation restricts the network in extracting pixel-level intrinsic structural or dependencies inside each patch, hurting the performance of pixel-level classification tasks. Methods: To tackle these issues, we propose a memory- and computation-efficient self-attention module to enable reasoning on relatively high-resolution features, promoting the efficiency of learning global information while effective grasping fine spatial details. Furthermore, we design a novel Multi-Branch Transformer (MultiTrans) architecture to provide hierarchical features for handling objects with variable shapes and sizes in medical images. By building four parallel Transformer branches on different levels of CNN, our hybrid network aggregates both multi-scale global contexts and multi-scale local features. Results: MultiTrans achieves the highest segmentation accuracy on three medical image datasets with different modalities: Synapse, ACDC and M&Ms. Compared to the Standard Self-Attention (SSA), the proposed Efficient Self-Attention (ESA) can largely reduce the training memory and computational complexity while even slightly improve the accuracy. Specifically, the training memory cost, FLOPs and Params of our ESA are 18.77%, 20.68% and 74.07% of the SSA. Conclusions: Experiments on three medical image datasets demonstrate the generality and robustness of the designed network. The ablation study shows the efficiency and effectiveness of our proposed ESA. Code is available at: https://github.com/Yanhua-Zhang/MultiTrans-extension .

HST-MRF: Heterogeneous Swin Transformer with Multi-Receptive Field for Medical Image Segmentation

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Mixed Transformer U-Net for Medical Image Segmentation

SSTrans-Net: Smart Swin Transformer Network for medical image segmentation

High-Resolution Swin Transformer for Automatic Medical Image Segmentation

Multi-dimension Transformer with Attention-based Filtering for Medical Image Segmentation

HCA-former: Hybrid Convolution Attention Transformer for 3D Medical Image Segmentation

SwinHR: Hemodynamic-powered hierarchical vision transformer for breast tumor segmentation

SMAFormer: Synergistic Multi-Attention Transformer for Medical Image Segmentation

Hybrid-Fusion Transformer for Multisequence MRI

ATTransUNet: an Enhanced Hybrid Transformer Architecture for Ultrasound and Histopathology Image Segmentation

HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation

MR-Trans: MultiResolution Transformer for Medical Image Segmentation.

Multi-Level Swin Transformer Enabled Automatic Segmentation and Classification of Breast Metastases

MultiTrans: Multi-branch transformer network for medical image segmentation

H2Former: An Efficient Hierarchical Hybrid Transformer for Medical Image Segmentation

Clinical Knowledge-Based Hybrid Swin Transformer for Brain Tumor Segmentation

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation

DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation

STM-UNet: An Efficient U-shaped Architecture Based on Swin Transformer and Multi-scale MLP for Medical Image Segmentation

HCTNet: A hybrid CNN-transformer network for breast ultrasound image segmentation