Abstract:Extracting semantic information from very-high-resolution (VHR) aerial images is a prominent topic in the Earth observation research. An increasing number of different sensor platforms are appearing in remote sensing, each of which can provide corresponding multimodal supplemental or enhanced information, such as optical images, light detection and ranging (LiDAR) point clouds, infrared images, or inertial measurement unit (IMU) data. However, these current deep networks for LiDAR and VHR images have not fully utilized the complete potential of multimodal data. The stacked multimodal fusion network (MFNet) ignores the structural differences between the modalities and the manual statistical characteristics within the modalities. For multimodal remote sensing data and its corresponding carefully designed handcrafted features, we designed a novel deep MFNet that can use multimodal VHR aerial images and LiDAR data and the corresponding intramodal features, such as LiDAR-derived features [slope and normalized digital surface model (NDSM)] and imagery-derived features [infrared–red–green (IRRG), normalized difference vegetation index (NDVI), and difference of Gaussian (DoG)]. Technically, we introduce the attention mechanism and multimodal learning to adaptively fuse intermodal and intramodal features. Specifically, we designed a multimodal fusion mechanism, pyramid dilation blocks, and a multilevel feature fusion module. Through these modules, our network realized the adaptive fusion of multimodal features, improved the receptive field, and enhanced the global-to-local contextual fusion effect. Moreover, we used a multiscale supervision training scheme to optimize the network. Extensive experimental results and ablation studies on the ISPRS semantic dataset and IEEE GRSS DFC Zeebrugge dataset show the effectiveness of our proposed MFNet.

Image semantic segmentation with hierarchical feature fusion based on deep neural network

Image Semantic Segmentation Based on Region and Deep Residual Network

Real-time Semantic Segmentation with Weighted Factorized-Depthwise Convolution

ISDNet: Integrating Shallow and Deep Networks for Efficient Ultra-high Resolution Segmentation

Research of improving semantic image segmentation based on a feature fusion model

Deep Feature Selection-And-Fusion for RGB-D Semantic Segmentation

Semantic Segmentation Network Based on Adaptive Attention and Deep Fusion Utilizing a Multi-Scale Dilated Convolutional Pyramid

A Deep Fully Convolution Neural Network for Semantic Segmentation Based on Adaptive Feature Fusion

Fusion of hierarchical class graphs for remote sensing semantic segmentation

HSNet: an Intelligent Hierarchical Semantic-Aware Network System for Real-Time Semantic Segmentation

Semantic-Aware Fusion Network Based on Super-Resolution

Remote Sensing Image Semantic Segmentation Method Based on a Deep Convolutional Neural Network and Multiscale Feature Fusion

Feature Fusion Network Based on Hybrid Attention for Semantic Segmentation

Semantic Image Segmentation with Improved Position Attention and Feature Fusion

Incorporating Luminance, Depth and Color Information by a Fusion-based Network for Semantic Segmentation

Hierarchical Self-Attention Embedded Neural Network With Dense Connection for Remote-Sensing Image Semantic Segmentation

MFEAFN: Multi-scale feature enhanced adaptive fusion network for image semantic segmentation

Research on Image Semantic Segmentation Based on Hybrid Cascade Feature Fusion and Detailed Attention Mechanism

Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network

Efficient Depth Fusion Transformer for Aerial Image Semantic Segmentation

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data