Abstract:Automated extraction of buildings from Earth observation (EO) data is important for various applications, including updating of maps, risk assessment, urban planning, and policy-making. Combining data from different sensors, such as high-resolution multispectral images (HRI) and light detection and ranging (LiDAR) data, has shown great potential in building extraction. Deep learning (DL) is increasingly used in multi-modal data fusion and urban object extraction. However, DL-based multi-modal fusion networks may under-perform due to insufficient learning of "joint features" from multiple sources and oversimplified approaches to fusing multi-modal features. Recently, a hybrid attention-aware fusion network (HAFNet) has been proposed for building extraction from a dataset, including co-located Very-High-Resolution (VHR) optical images and light detection and ranging (LiDAR) joint data. The system reported good performances thanks to the adaptivity of the attention mechanism to the features of the information content of the three streams but suffered from model over-parametrization, which inevitably leads to long training times and heavy computational load. In this paper, the authors propose a restructuring of the scheme, which involved replacing VGG-16-like encoders with the recently proposed EfficientNet, whose advantages counteract exactly the issues found with the HAFNet scheme. The novel configuration was tested on multiple benchmark datasets, reporting great improvements in terms of processing times, and also in terms of accuracy. The new scheme, called HAFNetE (HAFNet with EfficientNet integration), appears indeed capable of achieving good results with less parameters, translating into better computational efficiency. Based on these findings, we can conclude that, given the current advancements in single-thread schemes, the classical multi-thread HAFNet scheme could be effectively transformed by the HAFNetE scheme by replacing VGG-16 with EfficientNet blocks on each single thread. The remarkable reduction achieved in computational requirements moves the system one step closer to on-board implementation in a possible, future "urban mapping" satellite constellation.

EnergyNet: A modality-aware attention fusion network for building energy efficiency classification

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Enhancing buildings' energy efficiency prediction through advanced data fusion and fuzzy classification

Understanding Building Energy Efficiency with Administrative and Emerging Urban Big Data by Deep Learning in Glasgow

MVF-Net: A Multi-view Fusion Network for Event-based Object Classification

Integrating EfficientNet into an HAFNet Structure for Building Mapping in High-Resolution Optical Earth Observation Data

Energy Transformer

VAM-Net: Vegetation-Attentive Deep Network for Multi-modal Fusion of Visible-Light and Vegetation-Sensitive Images

LoadNet: enhancing energy storage system integration in power system operation using temporal convolutional and recurrent models with self-attention

Data fusion strategies for energy efficiency in buildings: Overview, challenges and novel orientations

Hybrid FusionNet: A Hybrid Feature Fusion Framework for Multisource High-Resolution Remote Sensing Image Classification

Energy-efficiency oriented occupancy space optimization in buildings: A data-driven approach based on multi-sensor fusion considering behavior-environment integration

M2FNet: Multi-modal Fusion Network for Object Detection from Visible and Thermal Infrared Images

A Hybrid Attention-Aware Fusion Network (HAFNet) for Building Extraction from High-Resolution Imagery and LiDAR Data

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

Ebbinghaus-Curve Guided Low-Rank Component-Induced Attention for Multisource Remote Sensing Classification

Feature-Decision Level Collaborative Fusion Network for Hyperspectral and LiDAR Classification

Advancing infrared and visible image fusion with an enhanced multiscale encoder and attention-based networks

Multilevel Attention Dynamic-Scale Network for HSI and LiDAR Data Fusion Classification

Attention based dual UNET network for infrared and visible image fusion