Abstract:Extracting semantic information from very-high-resolution (VHR) aerial images is a prominent topic in the Earth observation research. An increasing number of different sensor platforms are appearing in remote sensing, each of which can provide corresponding multimodal supplemental or enhanced information, such as optical images, light detection and ranging (LiDAR) point clouds, infrared images, or inertial measurement unit (IMU) data. However, these current deep networks for LiDAR and VHR images have not fully utilized the complete potential of multimodal data. The stacked multimodal fusion network (MFNet) ignores the structural differences between the modalities and the manual statistical characteristics within the modalities. For multimodal remote sensing data and its corresponding carefully designed handcrafted features, we designed a novel deep MFNet that can use multimodal VHR aerial images and LiDAR data and the corresponding intramodal features, such as LiDAR-derived features [slope and normalized digital surface model (NDSM)] and imagery-derived features [infrared–red–green (IRRG), normalized difference vegetation index (NDVI), and difference of Gaussian (DoG)]. Technically, we introduce the attention mechanism and multimodal learning to adaptively fuse intermodal and intramodal features. Specifically, we designed a multimodal fusion mechanism, pyramid dilation blocks, and a multilevel feature fusion module. Through these modules, our network realized the adaptive fusion of multimodal features, improved the receptive field, and enhanced the global-to-local contextual fusion effect. Moreover, we used a multiscale supervision training scheme to optimize the network. Extensive experimental results and ablation studies on the ISPRS semantic dataset and IEEE GRSS DFC Zeebrugge dataset show the effectiveness of our proposed MFNet.

Enhancing Multimodal Fusion with Only Unimodal Data

LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

Multicrop Fusion Strategy Based on Prototype Assignment for Remote Sensing Image Scene Classification.

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

MFFnet: Multimodal Feature Fusion Network for Synthetic Aperture Radar and Optical Image Land Cover Classification

Learning SAR-Optical Cross Modal Features for Land Cover Classification

MEFusion: Unsupervised Mutual Enhancement for Multimodal Image Fusion

A Spatial-Spectral Bilinear Representation Fusion Network for Multimodal Classification

Progressive fusion learning: A multimodal joint segmentation framework for building extraction from optical and SAR images

Multimodal Remote Sensing Data Classification Based on Gaussian Mixture Variational Dynamic Fusion Network

Few-Shot Remote Sensing Scene Classification with Multi-Metric Fusion

Multi-Resolution Multi-Modal Sensor Fusion For Remote Sensing Data With Label Uncertainty

Modality Fusion Vision Transformer for Hyperspectral and LiDAR Data Collaborative Classification

Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

Incomplete Multimodal Learning for Remote Sensing Data Fusion

Multimodal Fusion Method Based on Self-Attention Mechanism

Multimodal Frequeny Spectrum Fusion Schema for RGB-T Image Semantic Segmentation

MCFT: Multimodal Contrastive Fusion Transformer for Classification of Hyperspectral Image and LiDAR Data

Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation