Abstract:Multi-modal fusion overcomes the inherent limitations of single-sensor perception in 3D object detection of autonomous driving. The fusion of 4D Radar and LiDAR can boost the detection range and more robust. Nevertheless, different data characteristics and noise distributions between two sensors hinder performance improvement when directly integrating them. Therefore, we are the first to propose a novel fusion method termed $M^{2}$-Fusion for 4D Radar and LiDAR, based on Multi-modal and Multi-scale fusion. To better integrate two sensors, we propose an Interaction-based Multi-Modal Fusion (IMMF) method utilizing a self-attention mechanism to learn features from each modality and exchange intermediate layer information. Specific to the current single-resolution voxel division's precision and efficiency balance problem, we also put forward a Center-based Multi-Scale Fusion (CMSF) method to first regress the center points of objects and then extract features in multiple resolutions. Furthermore, we present a data preprocessing method based on Gaussian distribution that effectively decreases data noise to reduce errors caused by point cloud divergence of 4D Radar data in the $x$-$z$ plane. To evaluate the proposed fusion method, a series of experiments were conducted using the Astyx HiRes 2019 dataset, including the calibrated 4D Radar and 16-line LiDAR data. The results demonstrated that our fusion method compared favorably with state-of-the-art algorithms. When compared to PointPillars, our method achieves mAP (mean average precision) increases of 5.64$\%$ and 13.57$\%$ for 3D and BEV (bird's eye view) detection of the car class at a moderate level, respectively.

Camera-Radar Fusion Sensing System Based on Multi-Layer Perceptron

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Object Detection Using Multi-Sensor Fusion Based on Deep Learning

Multi-sensor Information Fusion for Unmanned Cars Using Radar Map

Study on Multi-Heterogeneous Sensor Data Fusion Method Based on Millimeter-Wave Radar and Camera

Radar and Camera Fusion Based Moving Obstacle Tracking for Automated Vehicles

A Multi-scale Fusion Obstacle Detection Algorithm for Autonomous Driving Based on Camera and Radar

An Interactive Multi Model prediction-based Radar-Camera Fusion Algorithm for Autonomous Vehicles

Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions

Research of Target Detection and Classification Techniques Using Millimeter-Wave Radar and Vision Sensors

SparseFusion3D: Sparse Sensor Fusion for 3D object detection by Radar and Camera in Environmental Perception

Radar Camera Fusion via Representation Learning in Autonomous Driving

Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles

Fusion Perception of Vision and Millimeter Wave Radar for Autonomous Driving

Data Fusion of Roadside Camera, LiDAR and Millimeter Wave Radar

Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving

Multi-Sensor Fusion Perception System in Train

Instance Fusion for Addressing Imbalanced Camera and Radar Data

A Deep Learning Method Based on the Multi-stage Fusion of Radar and Camera in UAV Obstacle Avoidance

Camera-Radar Fusion with Radar Channel Extension and Dual-CBAM-FPN for Object Detection