Abstract:In oncology research, accurate 3D segmentation of lesions from CT scans is essential for the modeling of lesion growth kinetics. However, following the RECIST criteria, radiologists routinely only delineate each lesion on the axial slice showing the largest transverse area, and delineate a small number of lesions in 3D for research purposes. As a result, we have plenty of unlabeled 3D volumes and labeled 2D images, and scarce labeled 3D volumes, which makes training a deep-learning 3D segmentation model a challenging task. In this work, we propose a novel model, denoted a multi-dimension unified Swin transformer (MDU-ST), for 3D lesion segmentation. The MDU-ST consists of a Shifted-window transformer (Swin-transformer) encoder and a convolutional neural network (CNN) decoder, allowing it to adapt to 2D and 3D inputs and learn the corresponding semantic information in the same encoder. Based on this model, we introduce a three-stage framework: 1) leveraging large amount of unlabeled 3D lesion volumes through self-supervised pretext tasks to learn the underlying pattern of lesion anatomy in the Swin-transformer encoder; 2) fine-tune the Swin-transformer encoder to perform 2D lesion segmentation with 2D RECIST slices to learn slice-level segmentation information; 3) further fine-tune the Swin-transformer encoder to perform 3D lesion segmentation with labeled 3D volumes. The network's performance is evaluated by the Dice similarity coefficient (DSC) and Hausdorff distance (HD) using an internal 3D lesion dataset with 593 lesions extracted from multiple anatomical locations. The proposed MDU-ST demonstrates significant improvement over the competing models. The proposed method can be used to conduct automated 3D lesion segmentation to assist radiomics and tumor growth modeling studies. This paper has been accepted by the IEEE International Symposium on Biomedical Imaging (ISBI) 2023.

Depth Swin Transformer Unet for Serial Section Biomedical Image Segmentation

TF-Unet:An Automatic Cardiac MRI Image Segmentation Method

DSTUNet: UNet with Efficient Dense SWIN Transformer Pathway for Medical Image Segmentation

DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation

DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation

Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation

Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images

SwinUNETR-V2: Stronger Swin Transformers with Stagewise Convolutions for 3D Medical Image Segmentation

Swin-TransUper: Swin Transformer-based UperNet for medical image segmentation

HiSwin UT: Hybrid Swin Transformers Network for Segmenting 3D in Vivo Two-photon Images of Rodent Cerebrovasculature

SSTrans-Net: Smart Swin Transformer Network for medical image segmentation

Multi-dimension unified Swin Transformer for 3D Lesion Segmentation in Multiple Anatomical Locations

STM-UNet: An Efficient U-shaped Architecture Based on Swin Transformer and Multi-scale MLP for Medical Image Segmentation

SwinBTS: A Method for 3D Multimodal Brain Tumor Segmentation Using Swin Transformer

DS-UNeXt: depthwise separable convolution network with large convolutional kernel for medical image segmentation

Swin Unet3D: a three-dimensional medical image segmentation network combining vision transformer and convolution

Going Beyond U-Net: Assessing Vision Transformers for Semantic Segmentation in Microscopy Image Analysis

ST-Unet: Swin Transformer boosted U-Net with Cross-Layer Feature Enhancement for medical image segmentation

3D Mitochondria Instance Segmentation with Spatio-Temporal Transformers

An Effective Fully Deep Convolutional Neural Network for Mitochondria Segmentation Based on ATUM-SEM.

DiffSwinTr: A diffusion model using 3D Swin Transformer for brain tumor segmentation