Abstract:Far-range perception through roadside sensors is crucial to the effectiveness of intelligent transportation systems. The main challenge of far-range perception is due to the difficulty of performing accurate object detection and tracking under far distances (e.g., > 150m) at a low cost. To cope with such challenges, deploying both millimeter wave Radars and high-definition (HD) cameras, and fusing their data for joint perception has become a common practice. The key to this solution, however, is the precise association between the two types of data, which are captured from different perspectives and have different degrees of measurement noises. Towards this goal, the first question is which plane to conduct the association, i.e., the 2D image plane or the BEV plane. We argue that the former is more suitable because the magnitude of location errors in the perspective projection points is smaller at far distances on the 2D plane and can lead to more accurate association. Thus, we first project the Radar-based target locations (on the BEV plane) to the 2D plane and then associate them with the camera-based object locations that are modeled as a point on each object. Subsequently, we map the camera-based object locations to the BEV plane through inverse projection mapping (IPM) with the corresponding depth information from the Radar data. Finally, we engage a BEV tracking module to generate target trajectories for traffic monitoring. Since our approach involves transformation between the 2D plane and BEV plane, we also devise a transformation parameters refining approach based on a depth scaling technique, utilising the above fusion process without requiring any additional devices such as GPS. We have deployed an actual testbed on an urban expressway and conducted extensive experiments to evaluate the effectiveness of our system. The results show that our system can improve AP BEV by 32%, and reduce the location error by 0.56m . Our system is capable of achieving an average location accuracy of 1.3m when we extend the detection range up to 500m . We thus believe that our proposed method offers a viable approach to efficient roadside far-range perception.

Fusion Perception of Vision and Millimeter Wave Radar for Autonomous Driving

Millimeter-Wave Radar and Vision Fusion Target Detection Algorithm Based on an Extended Network

Online Multipedestrian Tracking Based on Fused Detections of Millimeter Wave Radar and Vision

Online Multi-Object Tracking from A Bird's-Eye View by Fusion of Millimeter-Wave Radar and Vision

A Fusion Method Aiming at Environmental Perception of Autonomous Vehicle Based on Visual Scheme

Assisting the Visually Impaired: Multitarget Warning Through Millimeter Wave Radar and RGB-depth Sensors.

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Research of Target Detection and Classification Techniques Using Millimeter-Wave Radar and Vision Sensors

Study on Multi-Heterogeneous Sensor Data Fusion Method Based on Millimeter-Wave Radar and Camera

FARFusion: A Practical Roadside Radar-Camera Fusion System for Far-Range Perception

Target Detection Based on Information Fusion of Millimeter-Wave Radar and Visual Camera

Radar And Vision Fusion For The Real-Time Obstacle Detection And Identification

Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous Driving

Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions

Multitarget-Tracking Method Based on the Fusion of Millimeter-Wave Radar and LiDAR Sensor Information for Autonomous Vehicles

A Novel Multi-Sensor Fusion Based Object Detection and Recognition Algorithm for Intelligent Assisted Driving

Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar

Real-Time Target Detection System for Intelligent Vehicles Based on Multi-Source Data Fusion

Research on Vehicle Forward Target Recognition Algorithm Based on Vision and MMW Radar Fusion

Traffic Incident Detection Based on mmWave Radar and Improvement Using Fusion with Camera