Abstract:Medical image segmentation is crucial for healthcare, yet convolution-based methods like U-Net face limitations in modeling long-range dependencies. To address this, Transformers designed for sequence-to-sequence predictions have been integrated into medical image segmentation. However, a comprehensive understanding of Transformers' self-attention in U-Net components is lacking. TransUNet, first introduced in 2021, is widely recognized as one of the first models to integrate Transformer into medical image analysis. In this study, we present the versatile framework of TransUNet that encapsulates Transformers' self-attention into two key modules: (1) a Transformer encoder tokenizing image patches from a convolution neural network (CNN) feature map, facilitating global context extraction, and (2) a Transformer decoder refining candidate regions through cross-attention between proposals and U-Net features. These modules can be flexibly inserted into the U-Net backbone, resulting in three configurations: Encoder-only, Decoder-only, and Encoder+Decoder. TransUNet provides a library encompassing both 2D and 3D implementations, enabling users to easily tailor the chosen architecture. Our findings highlight the encoder's efficacy in modeling interactions among multiple abdominal organs and the decoder's strength in handling small targets like tumors. It excels in diverse medical applications, such as multi-organ segmentation, pancreatic tumor segmentation, and hepatic vessel segmentation. Notably, our TransUNet achieves a significant average Dice improvement of 1.06% and 4.30% for multi-organ segmentation and pancreatic tumor segmentation, respectively, when compared to the highly competitive nn-UNet, and surpasses the top-1 solution in the BrasTS2021 challenge. 2D/3D Code and models are available at https://github.com/Beckschen/TransUNet and https://github.com/Beckschen/TransUNet-3D , respectively.

Multi-scale Neighborhood Attention Transformer on U-Net for Medical Image Segmentation.

Mixed Transformer U-Net for Medical Image Segmentation

TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation.

MSCT-UNET: multi-scale contrastive transformer within U-shaped network for medical image segmentation

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

3D TransUNet: Advancing Medical Image Segmentation through Vision Transformers

A Multi-Scale Cross-Fusion Medical Image Segmentation Network Based on Dual-Attention Mechanism Transformer

HmsU-Net: A hybrid multi-scale U-net based on a CNN and transformer for medical image segmentation

H2MaT-Unet:Hierarchical hybrid multi-axis transformer based Unet for medical image segmentation

Stitching Transformer and Convolution in U-Net for Medical Image Segmentation

Contextual Attention Network: Transformer Meets U-Net

MSTCNet: Parallel Multi-Scale Network For Medical Image Segmentation.

MTC-TransUNet: A Multi-Scale Mixed Convolution TransUNet for Medical Image Segmentation

UNETR: Transformers for 3D Medical Image Segmentation

A novel full-convolution UNet-transformer for medical image segmentation

TSCA-Net: Transformer based spatial-channel attention segmentation network for medical images

UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation

MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet

DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation

TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers

U-Net Transformer: Self and Cross Attention for Medical Image Segmentation