Abstract:Semantic segmentation is an essential technique in remote sensing. Until recently, most related research has focused primarily on advancing semantic segmentation models based on monomodal imagery, and less attention has been given to models that utilize multimodal remote sensing data. Moreover, most current multimodal approaches consider only limited bimodal situations and cannot simultaneously utilize three or more modalities. The increase in expensive computational costs associated with previous feature fusion paradigms hinders their application in broader cases. How to design a unified method to cover a wide variety of quantity-agnostic modalities for multimodal semantic segmentation remains an unsolved issue. To address the aforementioned challenges, this study explores a feasible way and proposes a cost-effective multimodal sensing semantic segmentation model (MultiSenseSeg). MultiSenseSeg employs multiple lightweight modality-specific experts (MSEs), an adaptive multimodal matching (AMM) module, and a single feature extraction pipeline to efficiently model intramodal and intermodal relationships. Benefiting from these designs, the proposed MultiSenseSeg can serve as a unified multimodal model capable of addressing both monomodal and bimodal cases and readily extrapolating to scenarios with more modalities, thereby achieving semantic segmentation of arbitrary quantities of multimodal data. To evaluate the performance of our method, we select several state-of-the-art (SOTA) semantic segmentation models from the past three years and conduct extensive experiments on two public multimodal datasets. The results show that MultiSenseSeg can not only achieve higher accuracy but also exhibit user-friendly modality extrapolation, allowing end-to-end training for consumer-grade users based on limited hardware resources. The model's code will be available at https://github.com/W-qp/MultiSenseSeg.

Unsupervised Multimodal Feature Learning for Semantic Image Segmentation

Multi-Modal Unsupervised Feature Learning For Rgb-D Scene Labeling

Multimodal Contrastive Learning for Unpaired and Depth-privileged Semantic Segmentation.

Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Multi-Task Self-Supervised Learning for Image Segmentation Task

Multimodal Recurrent Neural Networks with Information Transfer Layers for Indoor Scene Labeling

SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular Images

General-Purpose Multimodal Transformer meets Remote Sensing Semantic Segmentation

MultiSenseSeg: A Cost-Effective Unified Multimodal Semantic Segmentation Model for Remote Sensing

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

Dual-Path Feature Fusion Network for Semantic Segmentation of Remote Sensing Images

Deep Feature Selection-And-Fusion for RGB-D Semantic Segmentation

Unsupervised Multi-Scale Hybrid Feature Extraction Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities

Multi-Modal Unsupervised Domain Adaptation for Semantic Image Segmentation

Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks

LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network

MMSS: Multi-modal Sharable and Specific Feature Learning for RGB-D Object Recognition.

Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning