Abstract:Geometric information in the normalized digital surface models (nDSM) is highly correlated with the semantic class of the land cover. Exploiting two modalities (RGB and nDSM (height)) jointly has great potential to improve the segmentation performance. However, it is still an under-explored field in remote sensing due to the following challenges. First, the scales of existing datasets are relatively small and the diversity of existing datasets is limited, which restricts the ability of validation. Second, there is a lack of unified benchmarks for performance assessment, which leads to difficulties in comparing the effectiveness of different models. Last, sophisticated multi-modal semantic segmentation methods have not been deeply explored for remote sensing data. To cope with these challenges, in this paper, we introduce a new remote-sensing benchmark dataset for multi-modal semantic segmentation based on RGB-Height (RGB-H) data. Towards a fair and comprehensive analysis of existing methods, the proposed benchmark consists of 1) a large-scale dataset including co-registered RGB and nDSM pairs and pixel-wise semantic labels; 2) a comprehensive evaluation and analysis of existing multi-modal fusion strategies for both convolutional and Transformer-based networks on remote sensing data. Furthermore, we propose a novel and effective Transformer-based intermediary multi-modal fusion (TIMF) module to improve the semantic segmentation performance through adaptive token-level multi-modal <a class="link-external link-http" href="http://fusion.The" rel="external noopener nofollow">this http URL</a> designed benchmark can foster future research on developing new methods for multi-modal learning on remote sensing data. Extensive analyses of those methods are conducted and valuable insights are provided through the experimental results. Code for the benchmark and baselines can be accessed at \url{<a class="link-external link-https" href="https://github.com/EarthNets/RSI-MMSegmentation" rel="external noopener nofollow">this https URL</a>}.

High-Resolution Multispectral Dataset for Semantic Segmentation

Algorithms for Semantic Segmentation of Multispectral Remote Sensing Imagery using Deep Learning

Low-Shot Learning for the Semantic Segmentation of Remote Sensing Imagery

Multispectral Semantic Segmentation for UAVs: A Benchmark Dataset and Baseline

A Semantic Segmentation Approach Based On Deeplab Network In High-Resolution Remote Sensing Images

EarthMapper: A Tool Box for the Semantic Segmentation of Remote Sensing Imagery

GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data

HS3-Bench: A Benchmark and Strong Baseline for Hyperspectral Semantic Segmentation in Driving Scenarios

Annotated Dataset for Training Cloud Segmentation Neural Networks Using High-Resolution Satellite Remote Sensing Imagery

Semantic Segmentation of Very High-Resolution Remote Sensing Image Based on Multiple Band Combinations and Patchwise Scene Analysis

AN END-TO-END FRAMEWORK FOR LOW-RESOLUTION REMOTE SENSING SEMANTIC SEGMENTATION

Object-Enhanced Semantic Segmentation Model for High-Resolution Remote Sensing Images

A Benchmark High-Resolution GaoFen-3 SAR Dataset for Building Semantic Segmentation

Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks

MLRSNet: A multi-label high spatial resolution remote sensing dataset for semantic scene understanding

Deep learning for semantic segmentation of remote sensing images with rich spectral content

Semantic Segmentation of Remote Sensing Images with Multi-scale Features and Attention Mechanism

Semantic Stereo for Incidental Satellite Images

DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation

One model is enough: Toward multiclass weakly supervised remote sensing image semantic segmentation