Abstract:Autonomous driving vehicles rely on sensors for the robust perception of their surroundings. Such vehicles are equipped with multiple perceptive sensors with a high level of redundancy to ensure safety and reliability in any driving condition. However, multi-sensor, such as camera, LiDAR, and radar systems raise requirements related to sensor calibration and synchronization, which are the fundamental blocks of any autonomous system. On the other hand, sensor fusion and integration have become important aspects of autonomous driving research and directly determine the efficiency and accuracy of advanced functions such as object detection and path planning. Classical model-based estimation and data-driven models are two mainstream approaches to achieving such integration. Most recent research is shifting to the latter, showing high robustness in real-world applications but requiring large quantities of data to be collected, synchronized, and properly categorized. However, there are two major research gaps in existing works: (i) they lack fusion (and synchronization) of multi-sensors, camera, LiDAR and radar; and (ii) generic scalable, and user-friendly end-to-end implementation. To generalize the implementation of the multi-sensor perceptive system, we introduce an end-to-end generic sensor dataset collection framework that includes both hardware deploying solutions and sensor fusion algorithms. The framework prototype integrates a diverse set of sensors, such as camera, LiDAR, and radar. Furthermore, we present a universal toolbox to calibrate and synchronize three types of sensors based on their characteristics. The framework also includes the fusion algorithms, which utilize the merits of three sensors, namely, camera, LiDAR, and radar, and fuse their sensory information in a manner that is helpful for object detection and tracking research. The generality of this framework makes it applicable in any robotic or autonomous applications and suitable for quick and large-scale practical deployment.

Robust multimodal perception in autonomous systems: A comprehensive review and enhancement strategies

Multimodal Sensors and ML‐Based Data Fusion for Advanced Robots

Emerging Trends in Autonomous Vehicle Perception: Multimodal Fusion for 3D Object Detection

Exploring the challenges and opportunities of image processing and sensor fusion in autonomous vehicles: A comprehensive review

Robust Cognitive Capability in Autonomous Driving Using Sensor Fusion Techniques: A Survey

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

Multi-Sensor Fusion and Cooperative Perception for Autonomous Driving: A Review

Autonomous Multi-Sensor Fusion Techniques for Environmental Perception in Self-Driving Vehicles

Environment Perception Technology for Intelligent Robots in Complex Environments: A Review

Multimodal Fusion for Robotics

Multimodal Semantic Segmentation in Autonomous Driving: A Review of Current Approaches and Future Perspectives

Collaborative Perception-The Missing Piece in Realizing Fully Autonomous Driving

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Advancing Autonomous Driving Perception: Analysis of Sensor Fusion and Computer Vision Techniques

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

A multimodal vision sensor for autonomous driving

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Robust Multi-Modal Sensor Fusion: An Adversarial Approach

Multimodal End-to-End Autonomous Driving

End-to-End Multimodal Sensor Dataset Collection Framework for Autonomous Vehicles