Abstract:Medical image segmentation plays a crucial role in advancing healthcare systems for disease diagnosis and treatment planning. The u-shaped architecture, popularly known as U-Net, has proven highly successful for various medical image segmentation tasks. However, U-Net's convolution-based operations inherently limit its ability to model long-range dependencies effectively. To address these limitations, researchers have turned to Transformers, renowned for their global self-attention mechanisms, as alternative architectures. One popular network is our previous TransUNet, which leverages Transformers' self-attention to complement U-Net's localized information with the global context. In this paper, we extend the 2D TransUNet architecture to a 3D network by building upon the state-of-the-art nnU-Net architecture, and fully exploring Transformers' potential in both the encoder and decoder design. We introduce two key components: 1) A Transformer encoder that tokenizes image patches from a convolution neural network (CNN) feature map, enabling the extraction of global contexts, and 2) A Transformer decoder that adaptively refines candidate regions by utilizing cross-attention between candidate proposals and U-Net features. Our investigations reveal that different medical tasks benefit from distinct architectural designs. The Transformer encoder excels in multi-organ segmentation, where the relationship among organs is crucial. On the other hand, the Transformer decoder proves more beneficial for dealing with small and challenging segmented targets such as tumor segmentation. Extensive experiments showcase the significant potential of integrating a Transformer-based encoder and decoder into the u-shaped medical image segmentation architecture. TransUNet outperforms competitors in various medical applications.

FDR-TransUNet: A novel encoder-decoder architecture with vision transformer for improved medical image segmentation

FTUNet: A Feature-Enhanced Network for Medical Image Segmentation Based on the Combination of U-Shaped Network and Vision Transformer

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

ViT-UperNet: a hybrid vision transformer with unified-perceptual-parsing network for medical image segmentation

Sfe-Transunet: A Transformer-Based U-Net With Skipped Features Enhancer For Medical Image Segmentation

A More Design-Flexible Medical Transformer for Volumetric Image Segmentation.

LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation

A novel full-convolution UNet-transformer for medical image segmentation

UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentation

FCTrans UNet: A Hybrid CNN and Transformer Model for Medical Image Segmentations

DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation

Trans-UNeter: A new Decoder of TransUNet for Medical Image Segmentation.

3D TransUNet: Advancing Medical Image Segmentation through Vision Transformers

A Novel Deep Learning Model for Medical Image Segmentation with Convolutional Neural Network and Transformer

Focal-UNet: UNet-like Focal Modulation for Medical Image Segmentation

A 3D Medical Image Segmentation Framework Fusing Convolution and Transformer Features

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

Context-aware and local-aware fusion with transformer for medical image segmentation

UNETR: Transformers for 3D Medical Image Segmentation

Medical Image Segmentation Using Dual Branch Networks with Embedded Attention Mechanism.

A Improved Spatial-Reduction Attention Transformer for Medical Image Segmentation