Abstract:Accurate semantic segmentation of remote sensing data plays a crucial role in the success of geoscience research and applications. Recently, multimodal fusion-based segmentation models have attracted much attention due to their outstanding performance as compared to conventional single-modal techniques. However, most of these models perform their fusion operation using convolutional neural networks (CNNs) or the vision transformer (Vit), resulting in insufficient local–global contextual modeling and representative capabilities. In this work, a multilevel multimodal fusion scheme called FTransUNet is proposed to provide a robust and effective multimodal fusion backbone for semantic segmentation by integrating both CNN and Vit into one unified fusion framework. First, the shallow-level features are first extracted and fused through convolutional layers and shallow-level feature fusion (SFF) modules. After that, deep-level features characterizing semantic information and spatial relationships are extracted and fused by a well-designed fusion Vit (FVit). It applies adaptively mutually boosted attention (Ada-MBA) layers and self-attention (SA) layers alternately in a three-stage scheme to learn cross-modality representations of high interclass separability and low intraclass variations. Specifically, the proposed Ada-MBA computes SA and cross-attention (CA) in parallel to enhance intra- and cross-modality contextual information simultaneously while steering attention distribution toward semantic-aware regions. As a result, FTransUNet can fuse shallow-level and deep-level features in a multilevel manner, taking full advantage of CNN and transformer to accurately characterize local details and global semantics, respectively. Extensive experiments confirm the superior performance of the proposed FTransUNet compared with other multimodal fusion approaches on two fine-resolution remote sensing datasets, namely ISPRS Vaihingen and Potsdam. The source code in this work is available at https://github.com/sstary/SSRS.

FuSC: Fusing Superpixels for Improved Semantic Consistency

How to Identify Good Superpixels for Deforestation Detection on Tropical Rainforests

Object-Level Semantic Segmentation on the High-Resolution Gaofen-3 FUSAR-Map Dataset

Research of improving semantic image segmentation based on a feature fusion model

Unsupervised segmentation via semantic-apparent feature fusion

Exploring Superpixel Segmentation Methods in the Context of Citizen Science and Deforestation Detection

Generating 10-Meter Resolution Land Use and Land Cover Products Using Historical Landsat Archive Based on Super Resolution Guided Semantic Segmentation Network

FUSU: A Multi-temporal-source Land Use Change Segmentation Dataset for Fine-grained Urban Semantic Understanding

Dual-Domain Fusion Network Based on Wavelet Frequency Decomposition and Fuzzy Spatial Constraint for Remote Sensing Image Segmentation

Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models

A2-FPN for Semantic Segmentation of Fine-Resolution Remotely Sensed Images

Semi-Automated Segmentation of Geoscientific Data Using Superpixels

A Semantic Segmentation-guided Approach for Ground-to-Aerial Image Matching

Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes

SFPNet: Sparse Focal Point Network for Semantic Segmentation on General LiDAR Point Clouds

A Prototypical Metric Learning Approach for Open-Set Semantic Segmentation on Remote Sensing Images

SIESEF-FusionNet: Spatial Inter-correlation Enhancement and Spatially-Embedded Feature Fusion Network for LiDAR Point Cloud Semantic Segmentation

Dual Super-Resolution Learning for Semantic Segmentation

A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation

Few-Shot Aerial Image Semantic Segmentation Leveraging Pyramid Correlation Fusion