Abstract:Perception of 3D remote sensing scenes plays a crucial role in accurately recognizing and locating ground objects, as it enables a deeper understanding of complex environments by capturing scene geometry, object relationships, and occlusion patterns. Inspired by the powerful multi-sensor fusion capabilities in autonomous driving, we explore a new task in this paper: given a set of multi-view images of a 3D remote sensing scene, we aim to obtain bird’s-eye view (BEV) scene information under the common view area in the world coordinate system. In this work, we focus on the task of semantic segmentation to demonstrate the feasibility of our approach and introduce a BEV modeling technique tailored for remote sensing scenes, which facilitates the projection of 3D scene details from multiple perspective views onto a bird’s-eye view. We then utilize a dual-encoder structure based on the Vision Transformer (VIT) architecture to extract relevant spatial information using self-attention mechanisms. Within the decoder, we employ a Feature Pyramid Network (FPN) to integrate BEV patch encoding with spatial feature residuals, enabling fine-grained segmentation results at the original input resolution. Furthermore, we curated the LEVIR-MDS multi-drone segmentation dataset, comprising scenes from 10 community-level areas across 3 continents, totaling 243k images and their corresponding annotated BEV semantic maps, amounting to approximately 500GB. This dataset serves as a robust benchmark to assess the effectiveness and generalization capability of our proposed method. To our best knowledge, this is the first semantic segmentation dataset designed specifically for collaborative multi-drone applications. We further show that our method achieves a 12% improvement in mIoU, reaching 69.73%, compared to a pure convolutional network model.

Cross-View Semantic Segmentation for Sensing Surroundings

Can We PASS Beyond the Field of View? Panoramic Annular Semantic Segmentation for Real-World Surrounding Perception

Unifying Terrain Awareness Through Real-Time Semantic Segmentation

In Defense Of Multi-Source Omni-Supervised Efficient Convnet For Robust Semantic Segmentation In Heterogeneous Unseen Domains

PASS: Panoramic Annular Semantic Segmentation

Omnisupervised Omnidirectional Semantic Segmentation

Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation

Robustifying Semantic Cognition of Traversability Across Wearable RGB-depth Cameras

Towards Robotic Semantic Segmentation of Supporting Surfaces

Real-time Semantic Segmentation in Traffic Scene Using Cross Stage Partial-based Encoder–decoder Network

Semantic perception of curbs beyond traversability for real-world navigation assistance systems

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

Active Scene Understanding via Online Semantic Reconstruction

Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation

CV-MOS: A Cross-View Model for Motion Segmentation

Learning Spatial-Semantic Features for Robust Video Object Segmentation

RSBEV: Multi-view Collaborative Segmentation of 3D Remote Sensing Scenes with Bird’s-Eye-View Representation

Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

Cross-view Transformers for real-time Map-view Semantic Segmentation

Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning

Semantic Translation With Convolutional Encoder-Decoder Networks For Viewpoint Estimation