Abstract:Sensor data fusion is essential for environmental perception within smart traffic applications. By using multiple sensors cooperatively, the accuracy and probability of the perception are increased, which is crucial for critical traffic scenarios or under bad weather conditions. In this paper, a modular real-time capable multi-sensor fusion framework is presented and tested to fuse data on the object list level from distributed automotive sensors (cameras, radar, and LiDAR). The modular multi-sensor fusion architecture receives an object list (untracked objects) from each sensor. The fusion framework combines classical data fusion algorithms, as it contains a coordinate transformation module, an object association module (Hungarian algorithm), an object tracking module (unscented Kalman filter), and a movement compensation module. Due to the modular design, the fusion framework is adaptable and does not rely on the number of sensors or their types. Moreover, the method continues to operate because of this adaptable design in case of an individual sensor failure. This is an essential feature for safety-critical applications. The architecture targets environmental perception in challenging time-critical applications. The developed fusion framework is tested using simulation and public domain experimental data. Using the developed framework, sensor fusion is obtained well below 10 milliseconds of computing time using an AMD Ryzen 7 5800H mobile processor and the Python programming language. Furthermore, the object-level multi-sensor approach enables the detection of changes in the extrinsic calibration of the sensors and potential sensor failures. A concept was developed to use the multi-sensor framework to identify sensor malfunctions. This feature will become extremely important in ensuring the functional safety of the sensors for autonomous driving.

Multi-Camera Object Fusion Tracking Model for Autonomous Driving.

Online Multi-Object Tracking from A Bird's-Eye View by Fusion of Millimeter-Wave Radar and Vision

Single-Camera and Inter-Camera Vehicle Tracking and 3D Speed Estimation Based on Fusion of Visual and Semantic Features

A Fusion Method Aiming at Environmental Perception of Autonomous Vehicle Based on Visual Scheme

3D Multiple Object Tracking with Multi-modal Fusion of Low-cost Sensors for Autonomous Driving.

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Real time object detection using LiDAR and camera fusion for autonomous driving

Real-time Multi-object Tracking Research Based on Image and Point Cloud Fusion

Multi-person Multi-Camera Tracking for Live Stream Videos Based on Improved Motion Model and Matching Cascade

Deep Learning-Based Robust Multi-Object Tracking via Fusion of mmWave Radar and Camera Sensors

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Joint Multi-Object Detection and Tracking with Camera-LiDAR Fusion for Autonomous Driving

Multiple-object Tracking Based on Monocular Camera and 3-D Lidar Fusion for Autonomous Vehicles

CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion

Research on multi-target tracking method based on multi-sensor fusion

Spatial-Temporal Correlation 3D Vehicle Detection and Tracking System with Multiple Surveillance Cameras

Multi-Sensor Data Fusion for Real-Time Multi-Object Tracking

Radar and Camera Fusion Based Moving Obstacle Tracking for Automated Vehicles

Object Detection Using Multi-Sensor Fusion Based on Deep Learning

Multi-3D-Object Tracking by Fusing RGB and 3D-LiDAR Data