Abstract:With the development of deep learning and remote sensing technologies in recent years, many semantic segmentation methods based on convolutional neural networks (CNNs) have been applied to road extraction. However, previous deep learning-based road extraction methods primarily used RGB imagery as an input and did not take advantage of the spectral information contained in hyperspectral imagery. These methods can produce discontinuous outputs caused by objects with similar spectral signatures to roads. In addition, the images obtained from different Earth remote sensing sensors may have different spatial resolutions, enhancing the difficulty of the joint analysis. This work proposes the Multiscale Fusion Attention Network (MSFANet) to overcome these problems. Compared to traditional road extraction frameworks, the proposed MSFANet fuses information from different spectra at multiple scales. In MSFANet, multispectral remote sensing data is used as an additional input to the network, in addition to RGB remote sensing data, to obtain richer spectral information. The Cross-source Feature Fusion Module (CFFM) is used to calibrate and fuse spectral features at different scales, reducing the impact of noise and redundant features from different inputs. The Multiscale Semantic Aggregation Decoder (MSAD) fuses multiscale features and global context information from the upsampling process layer by layer, reducing information loss during the multiscale feature fusion. The proposed MSFANet network was applied to the SpaceNet dataset and self-annotated images from Chongzhou, a representative city in China. Our MSFANet performs better over the baseline HRNet by a large margin of +6.38 IoU and +5.11 F1-score on the SpaceNet dataset, +3.61 IoU and +2.32 F1-score on the self-annotated dataset (Chongzhou dataset). Moreover, the effectiveness of MSFANet was also proven by comparative experiments with other studies.

CCAFFMNet: Dual-spectral semantic segmentation network with channel-coordinate attention feature fusion module

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic Segmentation

Attention-based Multi-modal Fusion Network for Semantic Scene Completion.

HAFFseg: RGB-Thermal semantic segmentation network with hybrid adaptive feature fusion strategy

IAFFNet: Illumination-Aware Feature Fusion Network for All-Day RGB-Thermal Semantic Segmentation of Road Scenes.

Rgb-t semantic segmentation based on cross-operational fusion attention in autonomous driving scenario

Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation

(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

Cross-modal Attention Fusion Network for RGB-D Semantic Segmentation

Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation

Semantic segmentation of remote sensing images based on dual‐channel attention mechanism

RGB and LiDAR Fusion-based 3D Semantic Segmentation for Autonomous Driving

AMCFNet: Asymmetric multiscale and crossmodal fusion network for RGB-D semantic segmentation in indoor service robots

Dual-branch deep cross-modal interaction network for semantic segmentation with thermal images

CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenes

Complementarity-aware cross-modal feature fusion network for RGB-T semantic segmentation

MSFANet: Multiscale Fusion Attention Network for Road Segmentation of Multispectral Remote Sensing Data

Dual Fusion Network for Hyperspectral Semantic Segmentation.

An Attention-Fused Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery

Dual-Path Multi-Level Feature Fusion Network for Semantic Segmentation of Remote Sensing Image