Abstract:Transformers have demonstrated significant promise for computer vision tasks. Particularly noteworthy is SwinUNETR, a model that employs vision transformers, which has made remarkable advancements in improving the process of segmenting medical images. Nevertheless, the efficacy of training process of SwinUNETR has been constrained by an extended training duration, a limitation primarily attributable to the integration of the attention mechanism within the architecture. In this article, to address this limitation, we introduce a novel framework, called the MetaSwin model. Drawing inspiration from the MetaFormer concept that uses other token mix operations, we propose a transformative modification by substituting attention-based components within SwinUNETR with a straightforward yet impactful spatial pooling operation. Additionally, we incorporate of Squeeze-and-Excitation (SE) blocks after each MetaSwin block of the encoder and into the decoder, which aims at segmentation performance. We evaluate our proposed MetaSwin model on two distinct medical datasets, namely BraTS 2023 and MICCAI 2015 BTCV, and conduct a comprehensive comparison with the two baselines, i.e. , SwinUNETR and SwinUNETR+SE models. Our results emphasize the effectiveness of MetaSwin, showcasing its competitive edge against the baselines, utilizing a simple pooling operation and efficient SE blocks. MetaSwin’s consistent and superior performance on the BTCV dataset, in comparison to SwinUNETR, is particularly significant. For instance, with a model size of 24, MetaSwin outperforms SwinUNETR’s 76.58% Dice score using fewer parameters (15,407,384 vs 15,703,304) and a substantially reduced training time (300 vs 467 mins), achieving an improved Dice score of 79.12%. This research highlights the essential contribution of a simplified transformer framework, incorporating basic elements such as pooling and SE blocks, thus emphasizing their potential to guide the progression of medical segmentation models, without relying on complex attention-based mechanisms.

Swin-UMamba†: Adapting Mamba-based Vision Foundation Models for Medical Image Segmentation

Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining

SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentation

U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation

Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentation

VM-UNET-V2 Rethinking Vision Mamba UNet for Medical Image Segmentation

Selective and Multi-Scale Fusion Mamba for Medical Image Segmentation

MetaSwin: a unified meta vision transformer model for medical image segmentation

HMT-UNet: A hybird Mamba-Transformer Vision UNet for Medical Image Segmentation

MedMamba: Vision Mamba for Medical Image Classification

SSTrans-Net: Smart Swin Transformer Network for medical image segmentation

HC-Mamba: Vision MAMBA with Hybrid Convolutional Techniques for Medical Image Segmentation

Rotate to Scan: UNet-like Mamba with Triplet SSM Module for Medical Image Segmentation

MambaClinix: Hierarchical Gated Convolution and Mamba-Based U-Net for Enhanced 3D Medical Image Segmentation

Tri-Plane Mamba: Efficiently Adapting Segment Anything Model for 3D Medical Images

LPAM: A lightweight medical segmentation network based on Mamba improved by prompt attention

SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation

VM-UNet: Vision Mamba UNet for Medical Image Segmentation