Abstract:Accurate semantic segmentation of remote sensing data plays a crucial role in the success of geoscience research and applications. Recently, multimodal fusion-based segmentation models have attracted much attention due to their outstanding performance as compared to conventional single-modal techniques. However, most of these models perform their fusion operation using convolutional neural networks (CNNs) or the vision transformer (Vit), resulting in insufficient local–global contextual modeling and representative capabilities. In this work, a multilevel multimodal fusion scheme called FTransUNet is proposed to provide a robust and effective multimodal fusion backbone for semantic segmentation by integrating both CNN and Vit into one unified fusion framework. First, the shallow-level features are first extracted and fused through convolutional layers and shallow-level feature fusion (SFF) modules. After that, deep-level features characterizing semantic information and spatial relationships are extracted and fused by a well-designed fusion Vit (FVit). It applies adaptively mutually boosted attention (Ada-MBA) layers and self-attention (SA) layers alternately in a three-stage scheme to learn cross-modality representations of high interclass separability and low intraclass variations. Specifically, the proposed Ada-MBA computes SA and cross-attention (CA) in parallel to enhance intra- and cross-modality contextual information simultaneously while steering attention distribution toward semantic-aware regions. As a result, FTransUNet can fuse shallow-level and deep-level features in a multilevel manner, taking full advantage of CNN and transformer to accurately characterize local details and global semantics, respectively. Extensive experiments confirm the superior performance of the proposed FTransUNet compared with other multimodal fusion approaches on two fine-resolution remote sensing datasets, namely ISPRS Vaihingen and Potsdam. The source code in this work is available at https://github.com/sstary/SSRS.

TransUNet with unified focal loss for class-imbalanced semantic segmentation

FCTrans UNet: A Hybrid CNN and Transformer Model for Medical Image Segmentations

Sfe-Transunet: A Transformer-Based U-Net With Skipped Features Enhancer For Medical Image Segmentation

FTUNet: A Feature-Enhanced Network for Medical Image Segmentation Based on the Combination of U-Shaped Network and Vision Transformer

TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation

Uncertainty teacher with dense focal loss for semi-supervised medical image segmentation

USCT-UNet: Rethinking the Semantic Gap in U-Net Network From U-Shaped Skip Connections With Multichannel Fusion Transformer

UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer

Enhancing Semantic Segmentation with Adaptive Focal Loss: A Novel Approach

DA-TransUNet: Integrating Spatial and Channel Dual Attention with Transformer U-Net for Medical Image Segmentation

A Weakly-Supervised Semantic Segmentation Approach Based on the Centroid Loss: Application to Quality Control and Inspection

IB-TransUNet: Combining Information Bottleneck and Transformer for Medical Image Segmentation

P-TransUNet: an improved parallel network for medical image segmentation

Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation

SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation

Medical Image Segmentation Based on TransUnet and A2B

TopPixelLoss: a Loss Function for Semantic Segmentation of Remote Sensing Images with Class Imbalance

CTMU-Net: an Improved U-Net for Semantic Segmentation of Remote-Sensing Images Based on the Combined Attention Mechanism

Unified semantic model for medical image segmentation

ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data