Abstract:Vision Transformer shows great superiority in medical image segmentation due to the ability in learning long-range dependency. For medical image segmentation from 3D data, such as computed tomography (CT), existing methods can be broadly classified into 2D-based and 3D-based methods. One key limitation in 2D-based methods is that the intra-slice information is ignored, while the limitation in 3D-based methods is the high computation cost and memory consumption, resulting in a limited feature representation for inner-slice information. During the clinical examination, radiologists primarily use the axial plane and then routinely review both axial and coronal planes to form a 3D understanding of anatomy. Motivated by this fact, our key insight is to design a hybrid model which can first learn fine-grained inner-slice information and then generate a 3D understanding of anatomy by incorporating 3D information. We present a novel \textbf{H}ybrid \textbf{Res}idual trans\textbf{Former} \textbf{(HResFormer)} for 3D medical image segmentation. Building upon standard 2D and 3D Transformer backbones, HResFormer involves two novel key designs: \textbf{(1)} a \textbf{H}ybrid \textbf{L}ocal-\textbf{G}lobal fusion \textbf{M}odule \textbf{(HLGM)} to effectively and adaptively fuse inner-slice information from 2D Transformer and intra-slice information from 3D volumes for 3D Transformer with local fine-grained and global long-range representation. \textbf{(2)} a residual learning of the hybrid model, which can effectively leverage the inner-slice and intra-slice information for better 3D understanding of anatomy. Experiments show that our HResFormer outperforms prior art on widely-used medical image segmentation benchmarks. This paper sheds light on an important but neglected way to design Transformers for 3D medical image segmentation.

Hybrid-Fusion Transformer for Multisequence MRI

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

HybridCTrm: Bridging CNN and Transformer for Multimodal Brain Image Segmentation

TransFusion: Multi-view Divergent Fusion for Medical Image Segmentation with Transformers

Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation.

Multi-Aperture Fusion of Transformer-Convolutional Network (MFTC-Net) for 3D Medical Image Segmentation and Visualization

TranSiam: Fusing Multimodal Visual Features Using Transformer for Medical Image Segmentation

HCA-former: Hybrid Convolution Attention Transformer for 3D Medical Image Segmentation

Context-aware and local-aware fusion with transformer for medical image segmentation

TransSea: Hybrid CNN–Transformer With Semantic Awareness for 3-D Brain Tumor Segmentation

TransSea: Hybrid CNN-Transformer with Semantic Awareness for 3D Brain Tumor Segmentation

A Unified Framework for Synthesizing Multisequence Brain MRI via Hybrid Fusion

MM-BiFPN: Multi-Modality Fusion Network With Bi-FPN for MRI Brain Tumor Segmentation

GSFormer: A Gated Skip Connection and Feature Fusion Transformer-based Neural Network for 3D MRI Brain Tumor Segmentation

A Parallelly Contextual Convolutional Transformer for Medical Image Segmentation

HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation

Hybrid Feature Fusion Neural Network Integrating Transformer for DCE-MRI Super Resolution

MMMViT: Multiscale multimodal vision transformer for brain tumor segmentation with missing modalities

The Fully Convolutional Transformer for Medical Image Segmentation