Abstract:Multi-camera-based simultaneous localization and mapping (SLAM) has been widely applied in various mobile robots under uncertain or unknown environments to accomplish tasks autonomously. However, the conventional purely data-driven feature extraction methods cannot utilize the rich semantic information in the environment, which leads to the performance of the SLAM system being susceptible to various interferences. In this work, we present a semantic-aware multi-level information fusion scheme for robust global orientation estimation. Specifically, a visual semantic perception system based on the synthesized surround view image is proposed for the multi-eye surround vision system widely used in mobile robots, which is used to obtain the visual semantic information required for SLAM tasks. The original multi-eye image was first transformed to the synthesized surround view image, and the passable space was extracted with the help of the semantic segmentation network model as a mask for feature extraction; moreover, the hybrid edge information was extracted to effectively eliminate the distorted edges by further using the distortion characteristics of the reverse perspective projection process. Then, the hybrid semantic information was used for robust global orientation estimation; thus, better localization performance was obtained. The experiments on an intelligent vehicle, which was used for automated valet parking both in indoor and outdoor scenes, showed that the proposed hybrid multi-level information fusion method achieved at least a 10-percent improvement in comparison with other edge segmentation methods, the average orientation estimation error being between 1 and 2 degrees, much smaller than other methods, and the trajectory drift value of the proposed method was much smaller than that of other methods.

Semantic-Aware Multi-modal Sensor Fusion for Motion Planning in Autonomous Driving

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Multi-Modal Neural Feature Fusion for Automatic Driving Through Perception-Aware Path Planning

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving

Multi-Modality Cascaded Fusion Technology for Autonomous Driving

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

Multi-Modal Sensor Fusion-Based Deep Neural Network for End-to-End Autonomous Driving With Scene Understanding

TLCFuse: Temporal Multi-Modality Fusion Towards Occlusion-Aware Semantic Segmentation-Aided Motion Planning

Multi-modal policy fusion for end-to-end autonomous driving

FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving

Robust 3D Semantic Segmentation Based on Multi-Phase Multi-Modal Fusion for Intelligent Vehicles

Enhanced Perception for Autonomous Driving Using Semantic and Geometric Data Fusion

Sensor Fusion by Spatial Encoding for Autonomous Driving

Multi-Sensor Fusion in Automated Driving: A Survey

Semantic-Structure-Aware Multi-Level Information Fusion for Robust Global Orientation Optimization of Autonomous Mobile Robots

Cognitive TransFuser: Semantics-guided Transformer-based Sensor Fusion for Improved Waypoint Prediction

Enhancing 3D object detection through multi-modal fusion for cooperative perception

Sensor Fusion: Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data

Multi-Sensor Fusion and Cooperative Perception for Autonomous Driving: A Review