Abstract:Medical image segmentation is crucial for healthcare, yet convolution-based methods like U-Net face limitations in modeling long-range dependencies. To address this, Transformers designed for sequence-to-sequence predictions have been integrated into medical image segmentation. However, a comprehensive understanding of Transformers' self-attention in U-Net components is lacking. TransUNet, first introduced in 2021, is widely recognized as one of the first models to integrate Transformer into medical image analysis. In this study, we present the versatile framework of TransUNet that encapsulates Transformers' self-attention into two key modules: (1) a Transformer encoder tokenizing image patches from a convolution neural network (CNN) feature map, facilitating global context extraction, and (2) a Transformer decoder refining candidate regions through cross-attention between proposals and U-Net features. These modules can be flexibly inserted into the U-Net backbone, resulting in three configurations: Encoder-only, Decoder-only, and Encoder+Decoder. TransUNet provides a library encompassing both 2D and 3D implementations, enabling users to easily tailor the chosen architecture. Our findings highlight the encoder's efficacy in modeling interactions among multiple abdominal organs and the decoder's strength in handling small targets like tumors. It excels in diverse medical applications, such as multi-organ segmentation, pancreatic tumor segmentation, and hepatic vessel segmentation. Notably, our TransUNet achieves a significant average Dice improvement of 1.06% and 4.30% for multi-organ segmentation and pancreatic tumor segmentation, respectively, when compared to the highly competitive nn-UNet, and surpasses the top-1 solution in the BrasTS2021 challenge. 2D/3D Code and models are available at https://github.com/Beckschen/TransUNet and https://github.com/Beckschen/TransUNet-3D , respectively.

MTC-TransUNet: A Multi-Scale Mixed Convolution TransUNet for Medical Image Segmentation

Mixed Transformer U-Net for Medical Image Segmentation

MSCT-UNET: multi-scale contrastive transformer within U-shaped network for medical image segmentation

FCTrans UNet: A Hybrid CNN and Transformer Model for Medical Image Segmentations

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

3D TransUNet: Advancing Medical Image Segmentation through Vision Transformers

TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers

HmsU-Net: A hybrid multi-scale U-net based on a CNN and transformer for medical image segmentation

HCT-Unet: multi-target medical image segmentation via a hybrid CNN-transformer Unet incorporating multi-axis gated multi-layer perceptron

A Novel Deep Learning Model for Medical Image Segmentation with Convolutional Neural Network and Transformer

IB-TransUNet: Combining Information Bottleneck and Transformer for Medical Image Segmentation

H2MaT-Unet:Hierarchical hybrid multi-axis transformer based Unet for medical image segmentation

MFLUnet: multi-scale fusion lightweight Unet for medical image segmentation

Sfe-Transunet: A Transformer-Based U-Net With Skipped Features Enhancer For Medical Image Segmentation

TransU²-Net: An Effective Medical Image Segmentation Framework Based on Transformer and U²-Net

A novel full-convolution UNet-transformer for medical image segmentation

EG-TransUNet: a transformer-based U-Net with enhanced and guided models for biomedical image segmentation

SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation

DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation

GCtx-UNet: Efficient Network for Medical Image Segmentation

MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet