Abstract:Accurate semantic segmentation of remote sensing data plays a crucial role in the success of geoscience research and applications. Recently, multimodal fusion-based segmentation models have attracted much attention due to their outstanding performance as compared to conventional single-modal techniques. However, most of these models perform their fusion operation using convolutional neural networks (CNNs) or the vision transformer (Vit), resulting in insufficient local–global contextual modeling and representative capabilities. In this work, a multilevel multimodal fusion scheme called FTransUNet is proposed to provide a robust and effective multimodal fusion backbone for semantic segmentation by integrating both CNN and Vit into one unified fusion framework. First, the shallow-level features are first extracted and fused through convolutional layers and shallow-level feature fusion (SFF) modules. After that, deep-level features characterizing semantic information and spatial relationships are extracted and fused by a well-designed fusion Vit (FVit). It applies adaptively mutually boosted attention (Ada-MBA) layers and self-attention (SA) layers alternately in a three-stage scheme to learn cross-modality representations of high interclass separability and low intraclass variations. Specifically, the proposed Ada-MBA computes SA and cross-attention (CA) in parallel to enhance intra- and cross-modality contextual information simultaneously while steering attention distribution toward semantic-aware regions. As a result, FTransUNet can fuse shallow-level and deep-level features in a multilevel manner, taking full advantage of CNN and transformer to accurately characterize local details and global semantics, respectively. Extensive experiments confirm the superior performance of the proposed FTransUNet compared with other multimodal fusion approaches on two fine-resolution remote sensing datasets, namely ISPRS Vaihingen and Potsdam. The source code in this work is available at https://github.com/sstary/SSRS.

Coastal Sea Fog Detection Based on Multi-channel Fusion Transformer

Multi-channel Satellite Cloud Image Fusion in the Tetrolet Transform Domain

Daytime Water Hazard Detection Based on Multi-Feature Fusion

An Image Segmentation Method Based on Transformer and Multi-Scale Feature Fusion for UAV Marine Environment Monitoring

Multi-source Data Fusion Network for Sea Fog Detection

TransFusion: Multi-Modal Robust Fusion for 3D Object Detection in Foggy Weather Based on Spatial Vision Transformer

Fog Detection over China’s Adjacent Sea Area by using the MTSAT Geostationary Satellite Data

Daytime Sea Fog Identification Based on Multi-Satellite Information and the ECA-TransUnet Model

A Transformer-based Multi-Modal Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Imagery

Sea fog detection based on unsupervised domain adaptation

Enhancement of Underwater Images through Parallel Fusion of Transformer and CNN

MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images

Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images

A fusion framework with multi-scale convolution and triple-branch cascaded transformer for underwater image enhancement

A Correlation Context-Driven Method for Sea Fog Detection in Meteorological Satellite Imagery

Hybrid Attention Fusion Embedded in Transformer for Remote Sensing Image Semantic Segmentation

A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation

A Multi-Tiered Collaborative Network for Optical Remote Sensing Fine-Grained Ship Detection in Foggy Conditions

Classification of hyperspectral and LiDAR data by transformer-based enhancement

SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection

A multimodal hyper-fusion transformer for remote sensing image classification