Abstract:Visual perception is a crucial component of autonomous driving systems. Traditional approaches for autonomous driving visual perception often rely on single-modal methods, and semantic segmentation tasks are accomplished by inputting RGB images. However, for semantic segmentation tasks in autonomous driving visual perception, a more effective strategy involves leveraging multiple modalities, which is because different sensors of the autonomous driving system bring diverse information, and the complementary features among different modalities enhance the robustness of the semantic segmentation modal. Contrary to the intuitive belief that more modalities lead to better accuracy, our research reveals that adding modalities to traditional semantic segmentation models can sometimes decrease precision. Inspired by the residual thinking concept, we propose a multimodal visual perception model which is capable of maintaining or even improving accuracy with the addition of any modality. Our approach is straightforward, using RGB as the main branch and employing the same feature extraction backbone for other modal branches. The modals score module (MSM) evaluates channel and spatial scores of all modality features, measuring their importance for overall semantic segmentation. Subsequently, the modal branches provide additional features to the RGB main branch through the features complementary module (FCM). Leveraging the residual thinking concept further enhances the feature extraction capabilities of all the branches. Through extensive experiments, we derived several conclusions. The integration of certain modalities into traditional semantic segmentation models tends to result in a decline in segmentation accuracy. In contrast, our proposed simple and scalable multimodal model demonstrates the ability to maintain segmentation precision when accommodating any additional modality. Moreover, our approach surpasses some state-of-the-art multimodal semantic segmentation models. Additionally, we conducted ablation experiments on the proposed model, confirming that the application of the proposed MSM, FCM, and the incorporation of residual thinking contribute significantly to the enhancement of the model.

Scatterer Recognition for Multi-Modal Intelligent Vehicular Channel Modeling via Synesthesia of Machines

Multi-Modal Intelligent Channel Modeling: A New Modeling Paradigm via Synesthesia of Machines

A LiDAR-Aided Channel Model for Vehicular Intelligent Sensing-Communication Integration

Multi-Feature Fusion Based Recognition and Relevance Analysis of Propagation Scenes for High-Speed Railway Channels

Slight-Distributed-Scatterer-Based Time-Varying Channel Modeling for Vehicular Environments

Connected Vehicle Channels: on the Consideration of Electromagnetic Scattering from Local Scatterers

A Semantic Communication Approach for Multiscene Target Detection in Intelligent Vehicle Networks

A New Sensing Channel Modeling Approach Based on Ray-Tracing and Stochastic Methods for Vehicle-to-Everything Applications

Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

Three-Dimensional Vehicle-to-Vehicle Channel Modeling with Multiple Moving Scatterers

A Novel Semantic Information Perception Architecture for Extreme Targets Detection in Complex Traffic Scenarios

3-D V2V MIMO CHANNEL MODELING IN DIFFERENT ROADWAY SCENARIOS WITH MOVING SCATTERERS

V2X-AHD:Vehicle-to-Everything Cooperation Perception via Asymmetric Heterogenous Distillation Network

A Unified Spatio-Temporal Description Model of Environment for Intelligent Vehicles

Multimodal LLM for Intelligent Transportation Systems

Synesthesia of Machines (SoM)-Enhanced Wideband Multi-User CSI Learning With LiDAR Sensing

Machine-Learning-Based Data Processing Techniques for Vehicle-to-Vehicle Channel Modeling

On the Modeling of Near-Field Scattering of Vehicles in Vehicle-to-X Wireless Channels Based on Scattering Centers

Simple Scalable Multimodal Semantic Segmentation Model

A Local Environment Model Based on Multi-Sensor Perception for Intelligent Vehicles