TEC-Net: Vision Transformer Embrace Convolutional Neural Networks for Medical Image Segmentation

Rui Sun,Tao Lei,Weichuan Zhang,Yong Wan,Yong Xia,Asoke K. Nandi

2023-12-20

Abstract:The hybrid architecture of convolution neural networks (CNN) and Transformer has been the most popular method for medical image segmentation. However, the existing networks based on the hybrid architecture suffer from two problems. First, although the CNN branch can capture image local features by using convolution operation, the vanilla convolution is unable to achieve adaptive extraction of image features. Second, although the Transformer branch can model the global information of images, the conventional self-attention only focuses on the spatial self-attention of images and ignores the channel and cross-dimensional self-attention leading to low segmentation accuracy for medical images with complex backgrounds. To solve these problems, we propose vision Transformer embrace convolutional neural networks for medical image segmentation (TEC-Net). Our network has two advantages. First, dynamic deformable convolution (DDConv) is designed in the CNN branch, which not only overcomes the difficulty of adaptive feature extraction using fixed-size convolution kernels, but also solves the defect that different inputs share the same convolution kernel parameters, effectively improving the feature expression ability of CNN branch. Second, in the Transformer branch, a (shifted)-window adaptive complementary attention module ((S)W-ACAM) and compact convolutional projection are designed to enable the network to fully learn the cross-dimensional long-range dependency of medical images with few parameters and calculations. Experimental results show that the proposed TEC-Net provides better medical image segmentation results than SOTA methods including CNN and Transformer networks. In addition, our TEC-Net requires fewer parameters and computational costs and does not rely on pre-training. The code is publicly available at <a class="link-external link-https" href="https://github.com/SR0920/TEC-Net" rel="external noopener nofollow">this https URL</a>.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address two main issues in the field of medical image segmentation:

1. **Limitations of Convolutional Neural Networks (CNNs) in feature extraction**: Traditional convolution operations can capture local features of images, but their fixed kernel size cannot achieve adaptive feature extraction. Moreover, different inputs share the same convolution kernel parameters, which limits the ability of CNNs to efficiently express features in medical images with complex backgrounds.

2. **Insufficiencies of Transformers in network attention mechanisms**: Although Transformers can model the global information of images, the traditional self-attention mechanism only focuses on spatial self-attention, ignoring channel and cross-dimensional self-attention. This results in lower segmentation accuracy for medical images with complex backgrounds.

To address these issues, the paper proposes a new method called "Transformer Embracing Convolutional Neural Network" (TEC-Net). TEC-Net has the following two key advantages:

- **Dynamic Deformable Convolution (DDConv)**: Designed in the CNN branch, dynamic deformable convolution not only overcomes the difficulty of adaptive feature extraction using fixed-size convolution kernels but also solves the problem of different inputs sharing the same convolution kernel parameters, effectively enhancing the feature expression capability of the CNN branch.

- **(Shifted)-Window Adaptive Complementary Attention Module ((S)W-ACAM)**: A new self-attention module designed in the Transformer branch, which learns weight coefficients adaptively through four parallel branches. This allows the network to fully learn the cross-dimensional long-range dependencies of medical images while reducing the number of parameters and computational cost.

Experimental results show that the proposed TEC-Net provides better segmentation performance on medical image segmentation tasks compared to other state-of-the-art methods (including pure CNN and pure Transformer networks), with fewer parameters and computational cost, and without the need for pre-training. Additionally, the code for this method has been publicly released.

In summary, TEC-Net aims to improve the performance of medical image segmentation by combining the advantages of CNNs and Transformers, especially when dealing with complex backgrounds or targets with subtle variations.