Abstract:Extracting semantic information from very-high-resolution (VHR) aerial images is a prominent topic in the Earth observation research. An increasing number of different sensor platforms are appearing in remote sensing, each of which can provide corresponding multimodal supplemental or enhanced information, such as optical images, light detection and ranging (LiDAR) point clouds, infrared images, or inertial measurement unit (IMU) data. However, these current deep networks for LiDAR and VHR images have not fully utilized the complete potential of multimodal data. The stacked multimodal fusion network (MFNet) ignores the structural differences between the modalities and the manual statistical characteristics within the modalities. For multimodal remote sensing data and its corresponding carefully designed handcrafted features, we designed a novel deep MFNet that can use multimodal VHR aerial images and LiDAR data and the corresponding intramodal features, such as LiDAR-derived features [slope and normalized digital surface model (NDSM)] and imagery-derived features [infrared–red–green (IRRG), normalized difference vegetation index (NDVI), and difference of Gaussian (DoG)]. Technically, we introduce the attention mechanism and multimodal learning to adaptively fuse intermodal and intramodal features. Specifically, we designed a multimodal fusion mechanism, pyramid dilation blocks, and a multilevel feature fusion module. Through these modules, our network realized the adaptive fusion of multimodal features, improved the receptive field, and enhanced the global-to-local contextual fusion effect. Moreover, we used a multiscale supervision training scheme to optimize the network. Extensive experimental results and ablation studies on the ISPRS semantic dataset and IEEE GRSS DFC Zeebrugge dataset show the effectiveness of our proposed MFNet.

Deep Image Annotation and Classification by Fusing Multi-Modal Semantic Topics

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

A Probabilistic Semantic Model for Image Annotation and Multi-Modal Image Retrieval

Deep Multimodal Data Fusion

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

Multi-Modal Multi-Label Semantic Indexing Of Images Based On Hybrid Ensemble Learning

Joint Semantic Segmentation using representations of LiDAR point clouds and camera images

CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion

Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images

Dual-attention-based semantic-aware self-supervised monocular depth estimation

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

Improving Fine-grained Image Classification with Multimodal Information

Multi-modal Deep Analysis for Multimedia

Towards Multi-Semantic Image Annotation with Graph Regularized Exclusive Group Lasso

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection

Transformer-Based Cross-Modal Information Fusion Network for Semantic Segmentation

A Deep and Autoregressive Approach for Topic Modeling of Multimodal Data

A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing

Multi-Modal Fusion-Based Multi-Task Semantic Communication System

Deep discriminative image feature learning for cross-modal semantics understanding