Abstract:Geometric information in the normalized digital surface models (nDSM) is highly correlated with the semantic class of the land cover. Exploiting two modalities (RGB and nDSM (height)) jointly has great potential to improve the segmentation performance. However, it is still an under-explored field in remote sensing due to the following challenges. First, the scales of existing datasets are relatively small and the diversity of existing datasets is limited, which restricts the ability of validation. Second, there is a lack of unified benchmarks for performance assessment, which leads to difficulties in comparing the effectiveness of different models. Last, sophisticated multi-modal semantic segmentation methods have not been deeply explored for remote sensing data. To cope with these challenges, in this paper, we introduce a new remote-sensing benchmark dataset for multi-modal semantic segmentation based on RGB-Height (RGB-H) data. Towards a fair and comprehensive analysis of existing methods, the proposed benchmark consists of 1) a large-scale dataset including co-registered RGB and nDSM pairs and pixel-wise semantic labels; 2) a comprehensive evaluation and analysis of existing multi-modal fusion strategies for both convolutional and Transformer-based networks on remote sensing data. Furthermore, we propose a novel and effective Transformer-based intermediary multi-modal fusion (TIMF) module to improve the semantic segmentation performance through adaptive token-level multi-modal <a class="link-external link-http" href="http://fusion.The" rel="external noopener nofollow">this http URL</a> designed benchmark can foster future research on developing new methods for multi-modal learning on remote sensing data. Extensive analyses of those methods are conducted and valuable insights are provided through the experimental results. Code for the benchmark and baselines can be accessed at \url{<a class="link-external link-https" href="https://github.com/EarthNets/RSI-MMSegmentation" rel="external noopener nofollow">this https URL</a>}.

SAT-MTB-SOS: A Benchmark Dataset for Satellite Video Single Object Segmentation

SatSOT: A Benchmark Dataset for Satellite Video Single Object Tracking

A Multitask Benchmark Dataset for Satellite Video: Object Detection, Tracking, and Segmentation.

Detecting and Tracking Small and Dense Moving Objects in Satellite Videos: A Benchmark

Satellite video single object tracking: A systematic review and an oriented object tracking benchmark

BSVOS: Background Interference Suppression Strategy for Satellite Video Multiobject Segmentation

YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark

Satellite Video Multi-Label Scene Classification With Spatial and Temporal Feature Cooperative Encoding: A Benchmark Dataset and Method

RSSOD-Bench: A large-scale benchmark dataset for Salient Object Detection in Optical Remote Sensing Imagery

The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking

State-Aware Tracker for Real-Time Video Object Segmentation.

LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking

Single Object Tracking in Satellite Videos Based on Feature Enhancement and Multi-Level Matching Strategy

The Unmanned Aerial Vehicle Benchmark: Object Detection, Tracking and Baseline

Object Tracking Based on Satellite Videos: A Literature Review

LaSOT: A High-quality Large-scale Single Object Tracking Benchmark

LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation

GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data

High-resolution satellite video single object tracking based on thicksiam framework

Vehicle Perception from Satellite

SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos