Abstract:Geometric information in the normalized digital surface models (nDSM) is highly correlated with the semantic class of the land cover. Exploiting two modalities (RGB and nDSM (height)) jointly has great potential to improve the segmentation performance. However, it is still an under-explored field in remote sensing due to the following challenges. First, the scales of existing datasets are relatively small and the diversity of existing datasets is limited, which restricts the ability of validation. Second, there is a lack of unified benchmarks for performance assessment, which leads to difficulties in comparing the effectiveness of different models. Last, sophisticated multi-modal semantic segmentation methods have not been deeply explored for remote sensing data. To cope with these challenges, in this paper, we introduce a new remote-sensing benchmark dataset for multi-modal semantic segmentation based on RGB-Height (RGB-H) data. Towards a fair and comprehensive analysis of existing methods, the proposed benchmark consists of 1) a large-scale dataset including co-registered RGB and nDSM pairs and pixel-wise semantic labels; 2) a comprehensive evaluation and analysis of existing multi-modal fusion strategies for both convolutional and Transformer-based networks on remote sensing data. Furthermore, we propose a novel and effective Transformer-based intermediary multi-modal fusion (TIMF) module to improve the semantic segmentation performance through adaptive token-level multi-modal <a class="link-external link-http" href="http://fusion.The" rel="external noopener nofollow">this http URL</a> designed benchmark can foster future research on developing new methods for multi-modal learning on remote sensing data. Extensive analyses of those methods are conducted and valuable insights are provided through the experimental results. Code for the benchmark and baselines can be accessed at \url{<a class="link-external link-https" href="https://github.com/EarthNets/RSI-MMSegmentation" rel="external noopener nofollow">this https URL</a>}.

Two novel benchmark datasets from ArcGIS and bing world imagery for remote sensing image retrieval

PatternNet: A benchmark dataset for performance evaluation of remote sensing image retrieval

RSI-CB: A Large-Scale Remote Sensing Image Classification Benchmark Using Crowdsourced Data

RSI-CB: A Large Scale Remote Sensing Image Classification Benchmark via Crowdsource Data

On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances, and Million-AID

Remote-Sensing Cross-Domain Scene Classification: A Dataset and Benchmark

Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark

Image retrieval from remote sensing big data: A survey

VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

GeoImageNet: a multi-source natural feature benchmark dataset for GeoAI and supervised machine learning

Performance Evaluation of Single-Label and Multi-Label Remote Sensing Image Retrieval Using a Dense Labeling Dataset

Multimodal remote sensing benchmark datasets for land cover classification with a shared and specific feature learning model

AI applications in forest monitoring need remote sensing benchmark datasets

Hashing for Retrieving Long-Tailed Distributed Remote Sensing Images

CartoMark: a benchmark dataset for map pattern recognition and map content retrieval with machine intelligence

Analysis of the inter-dataset representation ability of deep features for high spatial resolution remote sensing image scene classification

3MOS: Multi-sources, Multi-resolutions, and Multi-scenes dataset for Optical-SAR image matching

Csrs-Siat: A Benchmark Remote Sensing Dataset to Semantic-Enabled and Cross-Scales Scene Recognition

GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data

Remote Sensing Image Scene Classification: Benchmark and State of the Art