HGSFusion: Radar-Camera Fusion with Hybrid Generation and Synchronization for 3D Object Detection

Zijian Gu,Jianwei Ma,Yan Huang,Honghao Wei,Zhanye Chen,Hui Zhang,Wei Hong

2024-12-16

Abstract:Millimeter-wave radar plays a vital role in 3D object detection for autonomous driving due to its all-weather and all-lighting-condition capabilities for perception. However, radar point clouds suffer from pronounced sparsity and unavoidable angle estimation errors. To address these limitations, incorporating a camera may partially help mitigate the shortcomings. Nevertheless, the direct fusion of radar and camera data can lead to negative or even opposite effects due to the lack of depth information in images and low-quality image features under adverse lighting conditions. Hence, in this paper, we present the radar-camera fusion network with Hybrid Generation and Synchronization (HGSFusion), designed to better fuse radar potentials and image features for 3D object detection. Specifically, we propose the Radar Hybrid Generation Module (RHGM), which fully considers the Direction-Of-Arrival (DOA) estimation errors in radar signal processing. This module generates denser radar points through different Probability Density Functions (PDFs) with the assistance of semantic information. Meanwhile, we introduce the Dual Sync Module (DSM), comprising spatial sync and modality sync, to enhance image features with radar positional information and facilitate the fusion of distinct characteristics in different modalities. Extensive experiments demonstrate the effectiveness of our approach, outperforming the state-of-the-art methods in the VoD and TJ4DRadSet datasets by $6.53\%$ and $2.03\%$ in RoI AP and BEV AP, respectively. The code is available at <a class="link-external link-https" href="https://github.com/garfield-cpp/HGSFusion" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning,Robotics

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the problems of sparse point cloud and angular estimation error in millimeter - wave radar for 3D object detection. Specifically, millimeter - wave radar plays an important role in 3D object detection in autonomous driving due to its perception ability under all - weather and all - illumination conditions. However, radar point clouds have obvious sparsity and inevitable angular estimation errors, which limit their detection performance.

To overcome these problems, the authors propose a radar - camera fusion network named HGSFusion. Through the Hybrid Generation and Synchronization technology, it better fuses radar potential and image features to improve the accuracy of 3D object detection. The main problems to be solved include:

1. **Sparsity of radar point cloud**: Radar point clouds are usually sparser than LiDAR point clouds, resulting in a decline in detection performance.
2. **Angular estimation error**: The Direction of Arrival (DOA) estimation error in radar signal processing will lead to inaccurate point cloud distribution, further affecting the detection results.
3. **Challenges in multi - modal data fusion**: Direct fusion of radar and camera data may lead to negative effects because images lack depth information and the quality of image features is low under poor illumination conditions.

### Main contributions of HGSFusion

1. **Propose a new radar - camera fusion network HGSFusion** for enhancing the fusion of radar points and images.
2. **Radar Hybrid Generation Module (RHGM)**: Use point cloud distribution to generate denser and higher - quality radar point clouds, and reduce the impact of DOA estimation errors through different Probability Density Functions (PDFs).
3. **Dual Sync Module (DSM)**: By means of spatial synchronization and modal synchronization, use the position information of the radar to enhance image features and alleviate the impact of adverse illumination conditions on image features, thus generating fused BEV features.
4. **Experimental verification**: Extensive experiments on the VoD and TJ4DRadSet datasets show that this method outperforms the existing state - of - the - art methods by 6.53% and 2.03% in RoI AP and BEV AP respectively, verifying its effectiveness and robustness.

### Formula presentation

- **Probability density function of uniform distribution**:
\[
f_U(u, v)=\frac{1}{A}
\]
where \(A\) is the area of the uniform distribution.

- **Probability density function of Gaussian distribution**:
\[
f_G(u, v)=\frac{1}{2\pi b_1b_2}\exp\left(-\frac{1}{2}\left(\frac{(u - u_i)^2}{b_1^2}+\frac{(v - v_i)^2}{b_2^2}\right)\right)
\]
where \(b_1\) and \(b_2\) are standard deviations.

- **Mixed probability distribution**:
\[
f_H(u, v)=
\begin{cases}
f_G(u, v)&\text{if }(u, v)\in R_i(u, v)\\
f_U(u, v)&\text{if }(u, v)\in\complement_{R_m}R_i(u, v)\\
0&\text{if }(u, v)\notin R_m
\end{cases}
\]

Through these improvements, HGSFusion can achieve more accurate 3D object detection in complex environments, especially in cases where radar point clouds are sparse and angular estimation errors are large.