Abstract:Visual perception is a crucial component of autonomous driving systems. Traditional approaches for autonomous driving visual perception often rely on single-modal methods, and semantic segmentation tasks are accomplished by inputting RGB images. However, for semantic segmentation tasks in autonomous driving visual perception, a more effective strategy involves leveraging multiple modalities, which is because different sensors of the autonomous driving system bring diverse information, and the complementary features among different modalities enhance the robustness of the semantic segmentation modal. Contrary to the intuitive belief that more modalities lead to better accuracy, our research reveals that adding modalities to traditional semantic segmentation models can sometimes decrease precision. Inspired by the residual thinking concept, we propose a multimodal visual perception model which is capable of maintaining or even improving accuracy with the addition of any modality. Our approach is straightforward, using RGB as the main branch and employing the same feature extraction backbone for other modal branches. The modals score module (MSM) evaluates channel and spatial scores of all modality features, measuring their importance for overall semantic segmentation. Subsequently, the modal branches provide additional features to the RGB main branch through the features complementary module (FCM). Leveraging the residual thinking concept further enhances the feature extraction capabilities of all the branches. Through extensive experiments, we derived several conclusions. The integration of certain modalities into traditional semantic segmentation models tends to result in a decline in segmentation accuracy. In contrast, our proposed simple and scalable multimodal model demonstrates the ability to maintain segmentation precision when accommodating any additional modality. Moreover, our approach surpasses some state-of-the-art multimodal semantic segmentation models. Additionally, we conducted ablation experiments on the proposed model, confirming that the application of the proposed MSM, FCM, and the incorporation of residual thinking contribute significantly to the enhancement of the model.

Image Fusion Via Mutual Information Maximization for Semantic Segmentation in Autonomous Vehicles

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

Robust 3D Semantic Segmentation Based on Multi-Phase Multi-Modal Fusion for Intelligent Vehicles

Smooth fusion of multi-spectral images via total variation minimization for traffic scene semantic segmentation

Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

Infrared and Visible Image Fusion Via Mutual Information Maximization

Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation

Improved 3D Semantic Segmentation Model Based on RGB Image and LiDAR Point Cloud Fusion for Automantic Driving

Transformer-Based Cross-Modal Information Fusion Network for Semantic Segmentation

Accurate 3-D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multiview Edge Guidance and Fusion

A central multimodal fusion framework for outdoor scene image segmentation

FusionSAM: Latent Space driven Segment Anything Model for Multimodal Fusion and Segmentation

Simple Scalable Multimodal Semantic Segmentation Model

Enhancing Feature Fusion with Spatial Aggregation and Channel Fusion for Semantic Segmentation

Accurate 3D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multi-view Edge Guidance and Fusion

Semantic-Structure-Aware Multi-Level Information Fusion for Robust Global Orientation Optimization of Autonomous Mobile Robots

RGB and LiDAR Fusion-based 3D Semantic Segmentation for Autonomous Driving

LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

An efficient frequency domain fusion network of infrared and visible images

A RGB-Thermal Image Segmentation Method Based on Parameter Sharing and Attention Fusion for Safe Autonomous Driving