Abstract:Despite the rapid evolution of semantic segmentation for land cover classification in high-resolution remote sensing imagery, integrating multiple data modalities such as Digital Surface Model (DSM), RGB, and Near-infrared (NIR) remains a challenge. Current methods often process only two types of data, missing out on the rich information that additional modalities can provide. Addressing this gap, we propose a novel \textbf{L}ightweight \textbf{M}ultimodal data \textbf{F}usion \textbf{Net}work (LMFNet) to accomplish the tasks of fusion and semantic segmentation of multimodal remote sensing images. LMFNet uniquely accommodates various data types simultaneously, including RGB, NirRG, and DSM, through a weight-sharing, multi-branch vision transformer that minimizes parameter count while ensuring robust feature extraction. Our proposed multimodal fusion module integrates a \textit{Multimodal Feature Fusion Reconstruction Layer} and \textit{Multimodal Feature Self-Attention Fusion Layer}, which can reconstruct and fuse multimodal features. Extensive testing on public datasets such as US3D, ISPRS Potsdam, and ISPRS Vaihingen demonstrates the effectiveness of LMFNet. Specifically, it achieves a mean Intersection over Union ($mIoU$) of 85.09\% on the US3D dataset, marking a significant improvement over existing methods. Compared to unimodal approaches, LMFNet shows a 10\% enhancement in $mIoU$ with only a 0.5M increase in parameter count. Furthermore, against bimodal methods, our approach with trilateral inputs enhances $mIoU$ by 0.46 percentage points.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively fuse multiple data modalities (such as Digital Surface Model DSM, RGB, Near-Infrared NIR, etc.) in high-resolution remote sensing images to improve the performance of land cover classification tasks. Current methods typically can only handle two types of data, ignoring the rich information provided by other modalities. To solve this problem, the authors propose a lightweight multimodal data fusion network (LMFNet) aimed at achieving the fusion and semantic segmentation of multimodal remote sensing images. Specifically, the main contributions of the paper include: 1. Designing a novel Multimodal Feature Fusion Module (MFM) that can seamlessly fuse any number of modal data without modifying the structure of the fusion module. 2. Proposing a comprehensive multimodal fusion and semantic segmentation network LMFNet, including MFM and the preprocessing flow of multimodal data. 3. Conducting experiments on three public datasets to deeply analyze the internal fusion principles of the proposed fusion module and the details before and after multimodal feature fusion. The uniqueness of LMFNet lies in its ability to handle multiple data types (such as RGB, NirRG, and DSM) simultaneously and minimize the number of parameters through a multi-branch vision transformer with shared weights, ensuring strong feature extraction capabilities. Additionally, the multimodal fusion module in LMFNet includes a Multimodal Feature Fusion Reconstruction Layer and a Multimodal Feature Self-Attention Fusion Layer, capable of reconstructing and fusing multimodal features. Experimental results show that LMFNet achieved an 85.09% mean Intersection over Union (mIoU) on the US3D dataset, improving by 10% compared to single-modal methods, with only an additional 0.5M parameters. Compared to dual-modal methods, the LMFNet with three-modal input improved by 0.46 percentage points in mIoU. These results indicate that LMFNet not only has scalability and accuracy but also demonstrates efficiency in parameter usage, showing its broad application potential in remote sensing image analysis.

LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

CIMFNet: Cross-layer Interaction and Multiscale Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Images

A Transformer-based Multi-Modal Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Imagery

MFSA-Net: Semantic Segmentation with Camera-LiDAR Cross-Attention Fusion Based on Fast Neighbor Feature Aggregation

HR and LiDAR Data Collaborative Semantic Segmentation Based on Adaptive Cross-Modal Fusion Network

Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation

MFVNet: a deep adaptive fusion network with multiple field-of-views for remote sensing image semantic segmentation

Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data

Semantic Segmentation of Remote-Sensing Images Based on Multiscale Feature Fusion and Attention Refinement

MFCA-Net: a deep learning method for semantic segmentation of remote sensing images

SLMFNet: Enhancing land cover classification of remote sensing images through selective attentions and multi-level feature fusion

Lightweight Attention Network for Very High-Resolution Image Semantic Segmentation

FDGSNet: A Multi-modal Gated Segmentation Network for Remote Sensing Image Based on Frequency Decomposition

An Attention-Fused Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery

MFFNet: Multimodal Feature Fusion Network for Point Cloud Semantic Segmentation

Multi-View Feature Fusion and Rich Information Refinement Network for Semantic Segmentation of Remote Sensing Images

A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation

Multi-sensor Fusion Network for Unstructured Scene Segmentation with Surface Normal Incorporated