Abstract:Geometric information in the normalized digital surface models (nDSM) is highly correlated with the semantic class of the land cover. Exploiting two modalities (RGB and nDSM (height)) jointly has great potential to improve the segmentation performance. However, it is still an under-explored field in remote sensing due to the following challenges. First, the scales of existing datasets are relatively small and the diversity of existing datasets is limited, which restricts the ability of validation. Second, there is a lack of unified benchmarks for performance assessment, which leads to difficulties in comparing the effectiveness of different models. Last, sophisticated multi-modal semantic segmentation methods have not been deeply explored for remote sensing data. To cope with these challenges, in this paper, we introduce a new remote-sensing benchmark dataset for multi-modal semantic segmentation based on RGB-Height (RGB-H) data. Towards a fair and comprehensive analysis of existing methods, the proposed benchmark consists of 1) a large-scale dataset including co-registered RGB and nDSM pairs and pixel-wise semantic labels; 2) a comprehensive evaluation and analysis of existing multi-modal fusion strategies for both convolutional and Transformer-based networks on remote sensing data. Furthermore, we propose a novel and effective Transformer-based intermediary multi-modal fusion (TIMF) module to improve the semantic segmentation performance through adaptive token-level multi-modal <a class="link-external link-http" href="http://fusion.The" rel="external noopener nofollow">this http URL</a> designed benchmark can foster future research on developing new methods for multi-modal learning on remote sensing data. Extensive analyses of those methods are conducted and valuable insights are provided through the experimental results. Code for the benchmark and baselines can be accessed at \url{<a class="link-external link-https" href="https://github.com/EarthNets/RSI-MMSegmentation" rel="external noopener nofollow">this https URL</a>}.

A large-scale climate-aware satellite image dataset for domain adaptive land-cover semantic segmentation

Annotated Dataset for Training Cloud Segmentation Neural Networks Using High-Resolution Satellite Remote Sensing Imagery

So2Sat LCZ42: A Benchmark Dataset for Global Local Climate Zones Classification

Enabling Country-Scale Land Cover Mapping with Meter-Resolution Satellite Imagery

LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation

CAS Landslide Dataset: A Large-Scale and Multisensor Dataset for Deep Learning-Based Landslide Detection

Large-Scale Land Cover Mapping with Fine-Grained Classes Via Class-Aware Semi-Supervised Semantic Segmentation

iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images

SCDA: A Style and Content Domain Adaptive Semantic Segmentation Method for Remote Sensing Images

3D Semantic Segmentation in the Wild: Learning Generalized Models for Adverse-Condition Point Clouds

So2Sat LCZ42: A Benchmark Data Set for the Classification of Global Local Climate Zones [Software and Data Sets]

FUSU: A Multi-temporal-source Land Use Change Segmentation Dataset for Fine-grained Urban Semantic Understanding

DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation

Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks

Curriculum-style Local-to-global Adaptation for Cross-domain Remote Sensing Image Segmentation

GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data

SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery

Remote-Sensing Cross-Domain Scene Classification: A Dataset and Benchmark

Weather-aware autopilot: Domain generalization for point cloud semantic segmentation in diverse weather scenarios

An Unsupervised Domain Adaption Framework for Aerial Image Semantic Segmentation Based on Curriculum Learning

Satellite Video Multi-Label Scene Classification With Spatial and Temporal Feature Cooperative Encoding: A Benchmark Dataset and Method