Abstract:Visual perception is a crucial component of autonomous driving systems. Traditional approaches for autonomous driving visual perception often rely on single-modal methods, and semantic segmentation tasks are accomplished by inputting RGB images. However, for semantic segmentation tasks in autonomous driving visual perception, a more effective strategy involves leveraging multiple modalities, which is because different sensors of the autonomous driving system bring diverse information, and the complementary features among different modalities enhance the robustness of the semantic segmentation modal. Contrary to the intuitive belief that more modalities lead to better accuracy, our research reveals that adding modalities to traditional semantic segmentation models can sometimes decrease precision. Inspired by the residual thinking concept, we propose a multimodal visual perception model which is capable of maintaining or even improving accuracy with the addition of any modality. Our approach is straightforward, using RGB as the main branch and employing the same feature extraction backbone for other modal branches. The modals score module (MSM) evaluates channel and spatial scores of all modality features, measuring their importance for overall semantic segmentation. Subsequently, the modal branches provide additional features to the RGB main branch through the features complementary module (FCM). Leveraging the residual thinking concept further enhances the feature extraction capabilities of all the branches. Through extensive experiments, we derived several conclusions. The integration of certain modalities into traditional semantic segmentation models tends to result in a decline in segmentation accuracy. In contrast, our proposed simple and scalable multimodal model demonstrates the ability to maintain segmentation precision when accommodating any additional modality. Moreover, our approach surpasses some state-of-the-art multimodal semantic segmentation models. Additionally, we conducted ablation experiments on the proposed model, confirming that the application of the proposed MSM, FCM, and the incorporation of residual thinking contribute significantly to the enhancement of the model.

Multimodal Remote Sensing Image Segmentation With Intuition-Inspired Hypergraph Modeling

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Transformer-induced graph reasoning for multimodal semantic segmentation in remote sensing

MultiSenseSeg: A Cost-Effective Unified Multimodal Semantic Segmentation Model for Remote Sensing

Multi-modal remote sensing image segmentation based on attention-driven dual-branch encoding framework

A Cyclic Information–Interaction Model for Remote Sensing Image Segmentation

Simple Scalable Multimodal Semantic Segmentation Model

Multi-scale attention fusion network for semantic segmentation of remote sensing images

Multielement Feature-Based Hierarchical Context Integration Network for Remote Sensing Image Segmentation

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

A Synergistical Attention Model for Semantic Segmentation of Remote Sensing Images

Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation

CIMFNet: Cross-layer Interaction and Multiscale Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Images

General-Purpose Multimodal Transformer meets Remote Sensing Semantic Segmentation

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

Multi-Modal Prototypes for Open-World Semantic Segmentation

Object Segmentation by Mining Cross-Modal Semantics

A Transformer-based Multi-Modal Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Imagery

Dynamic High-Resolution Network for Semantic Segmentation in Remote-Sensing Images

A Multi-Step Fusion Network for Semantic Segmentation of High-Resolution Aerial Images

Learning Unified Hyper-Network for Multi-Modal MR Image Synthesis and Tumor Segmentation With Missing Modalities