Abstract:Sensor data fusion is essential for environmental perception within smart traffic applications. By using multiple sensors cooperatively, the accuracy and probability of the perception are increased, which is crucial for critical traffic scenarios or under bad weather conditions. In this paper, a modular real-time capable multi-sensor fusion framework is presented and tested to fuse data on the object list level from distributed automotive sensors (cameras, radar, and LiDAR). The modular multi-sensor fusion architecture receives an object list (untracked objects) from each sensor. The fusion framework combines classical data fusion algorithms, as it contains a coordinate transformation module, an object association module (Hungarian algorithm), an object tracking module (unscented Kalman filter), and a movement compensation module. Due to the modular design, the fusion framework is adaptable and does not rely on the number of sensors or their types. Moreover, the method continues to operate because of this adaptable design in case of an individual sensor failure. This is an essential feature for safety-critical applications. The architecture targets environmental perception in challenging time-critical applications. The developed fusion framework is tested using simulation and public domain experimental data. Using the developed framework, sensor fusion is obtained well below 10 milliseconds of computing time using an AMD Ryzen 7 5800H mobile processor and the Python programming language. Furthermore, the object-level multi-sensor approach enables the detection of changes in the extrinsic calibration of the sensors and potential sensor failures. A concept was developed to use the multi-sensor framework to identify sensor malfunctions. This feature will become extremely important in ensuring the functional safety of the sensors for autonomous driving.

MultiTest: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems

Multitest:Physical-Aware Object Insertion for Testing Multi-Sensor Fusion Perception Systems

Unity is Strength? Benchmarking the Robustness of Fusion-based 3D Object Detection Against Physical Sensor Attack

Online Multi-Object Tracking from A Bird's-Eye View by Fusion of Millimeter-Wave Radar and Vision

Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunities

The Design of a Multi-Sensor Tracking and Fusion Algorithms Test-Bed

Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar

Multisensor Data Fusion in Testability Evaluation of Equipment

3D Environmental Perception Modeling in the Simulated Autonomous-Driving Systems

Multi-Sensor Data Fusion for Real-Time Multi-Object Tracking

Design and implementation of algorithm-level testbed for multi-sensor information fusion

Multi-sensor Fusion for Interactive Visual Computing in Mixed Environment.

COMMIT: Certifying Robustness of Multi-Sensor Fusion Systems against Semantic Attacks

Multi-level and Metrics Evaluation Approach for Data-Driven Based Sensor Models

Detecting Multi-Sensor Fusion Errors in Advanced Driver-Assistance Systems

AFMCT: adaptive fusion module based on cross-modal transformer block for 3D object detection

Study on Multi-Heterogeneous Sensor Data Fusion Method Based on Millimeter-Wave Radar and Camera

External Joint Calibration of A Novel Multi-Modal Perception System

Multi-Sem Fusion: Multimodal Semantic Fusion for 3-D Object Detection

A Detachable and Expansible Multisensor Data Fusion Model for Perception in Level 3 Autonomous Driving System

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey