Abstract:Building extraction is significant in urban planning, economic evaluation, and driverless technology development. However, automatic building extraction from high spatial resolution remote sensing images has been a challenging task due to the various building shapes and colors, imaging conditions, and complex background objects. Current methods in building extraction are generally based on deep convolution networks, and they mostly use an encoder-decoder architecture, wherein detailed building features and small buildings are easily omitted in continuous convolution operations. Moreover, buildings with blurred boundaries are only completely extracted with difficulty. To meet these challenges, we propose a multi-task architecture of frequency-spatial learning Transformer to extract buildings from high spatial resolution remote sensing images. Different from current architecture, we designed a frequency-spatial learning module in the framework of multi-task to synthesize the multi-scale spatial features and frequency decomposition features of high-resolution image. Spiking convolution is proposed in this study to enhance the frequency features of buildings by mimicking the neural transmission in human brains. In this way, multi-scale building features can be better preserved and distinguished from background objects. Moreover, a masked-attention Transformer is adopted to improve multi-scale building mask prediction accuracy by synthesizing successive pixel-wise up-sampled feature maps. We also propose a strategy to evaluate the practical transferability of the proposed method by mimicking practical application cases through training and evaluating images with different spatial resolutions from different study areas and datasets. Experiments using five public building datasets (WHU-Building Satellite Dataset I, WHU-Building Satellite Dataset II, Massachusetts Buildings Dataset, Inria Aerial Image Dataset, xBD Building Dataset) demonstrate the strong potential applicability of our proposed method for practical application cases. Our method outperforms five recently proposed state-of-the-art semantic segmentation methods with 36.60% accuracy improvement on extracted buildings and approximately 53.55% recall progress in extracting small building instances. The implementation code will be released after the paper is published.

Data Fusion for Multi-Task Learning of Building Extraction and Height Estimation

FusionHeightNet: A Multi-Level Cross-Fusion Method from Multi-Source Remote Sensing Images for Urban Building Height Estimation

LIGHT: Joint Individual Building Extraction and Height Estimation from Satellite Images through a Unified Multitask Learning Network

A Multiscale and Multitask Deep Learning Framework for Automatic Building Extraction

Enhanced Radar Perception via Multi-Task Learning: Towards Refined Data for Sensor Fusion Applications

MF-BHNet: A Hybrid Multimodal Fusion Network for Building Height Estimation Using Sentinel-1 and Sentinel-2 Imagery

Multi-Task Multi-Sensor Fusion for 3D Object Detection

Large-Scale Fine-Grained Building Classification and Height Estimation for Semantic Urban Reconstruction: Outcome of the 2023 IEEE GRSS Data Fusion Contest

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Multi-task deep learning for large-scale building detail extraction from high-resolution satellite imagery

HGDNet: A Height-Hierarchy Guided Dual-Decoder Network for Single View Building Extraction and Height Estimation

An improved self-training network for building and road extraction in urban areas by integrating optical and radar remotely sensed data

MSFTrans: a multi-task frequency-spatial learning transformer for building extraction from high spatial resolution remote sensing images

Multi-Task Learning of Relative Height Estimation and Semantic Segmentation from Single Airborne RGB Images

Data Fusion of High-Resolution Satellite Imagery and GIS Data for Automatic Building Extraction

Multi-Task Learning of Height and Semantics from Aerial Images

AUTOMATIC BUILDING EXTRACTION FROM LIDAR POINT CLOUD DATA IN THE FUSION OF ORTHOIMAGE

Simultaneous Extraction of Spatial and Attributional Building Information Across Large-Scale Urban Landscapes from High-Resolution Satellite Imagery

Deep cross-domain building extraction for selective depth estimation from oblique aerial imagery

Progressive fusion learning: A multimodal joint segmentation framework for building extraction from optical and SAR images

A Generalized Multi-Task Learning Approach to Stereo DSM Filtering in Urban Areas