V2X-R: Cooperative LiDAR-4D Radar Fusion for 3D Object Detection with Denoising Diffusion

Xun Huang,Jinlong Wang,Qiming Xia,Siheng Chen,Bisheng Yang,Cheng Wang,Chenglu Wen

2024-11-13

Abstract:Current Vehicle-to-Everything (V2X) systems have significantly enhanced 3D object detection using LiDAR and camera data. However, these methods suffer from performance degradation in adverse weather conditions. The weatherrobust 4D radar provides Doppler and additional geometric information, raising the possibility of addressing this challenge. To this end, we present V2X-R, the first simulated V2X dataset incorporating LiDAR, camera, and 4D radar. V2X-R contains 12,079 scenarios with 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes. Subsequently, we propose a novel cooperative LiDAR-4D radar fusion pipeline for 3D object detection and implement it with various fusion strategies. To achieve weather-robust detection, we additionally propose a Multi-modal Denoising Diffusion (MDD) module in our fusion pipeline. MDD utilizes weather-robust 4D radar feature as a condition to prompt the diffusion model to denoise noisy LiDAR features. Experiments show that our LiDAR-4D radar fusion pipeline demonstrates superior performance in the V2X-R dataset. Over and above this, our MDD module further improved the performance of basic fusion model by up to 5.73%/6.70% in foggy/snowy conditions with barely disrupting normal performance. The dataset and code will be publicly available at: <a class="link-external link-https" href="https://github.com/ylwhxht/V2X-R" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the issue of performance degradation in current Vehicle-to-Everything (V2X) systems for 3D object detection under adverse weather conditions. Specifically, existing 3D object detection methods primarily rely on LiDAR and camera data, which are susceptible to interference in adverse weather (such as fog and snow), leading to significant performance drops. To tackle this challenge, the paper proposes the following key points:

1. **Introduction of 4D Radar**: 4D radar can provide Doppler information and additional geometric information, and it is robust against adverse weather. Therefore, the paper proposes to fuse 4D radar with LiDAR to improve the performance of 3D object detection under adverse weather conditions.

2. **Construction of V2X-R Dataset**: The paper constructs the first simulated V2X dataset, V2X-R, which includes LiDAR, camera, and 4D radar data. This dataset contains 12,079 scenes, 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes. This dataset provides a foundation for research on multimodal fusion.

3. **Proposing Multimodal Denoising Diffusion (MDD) Module**: To further improve detection performance under adverse weather conditions, the paper designs a Multimodal Denoising Diffusion (MDD) module. This module leverages the weather robustness of 4D radar as a condition to guide the diffusion model in removing noise from LiDAR features. Experimental results show that the MDD module improves the performance of the baseline fusion model by 5.73% and 6.70% under foggy and snowy conditions, respectively, with almost no impact on performance under normal weather conditions.

### Summary

By introducing 4D radar and constructing the V2X-R dataset, the paper proposes a new LiDAR-4D radar fusion pipeline and designs the MDD module to address noise issues under adverse weather conditions. These innovations not only enhance the performance of 3D object detection under adverse weather conditions but also provide important data and methodological support for future research on multimodal fusion.