Abstract:Perceiving the surrounding environment is a fundamental task in autonomous driving. To obtain highly accurate perception results, modern autonomous driving systems typically employ multi-modal sensors to collect comprehensive environmental data. Among these, the radar-camera multi-modal perception system is especially favored for its excellent sensing capabilities and cost-effectiveness. However, the substantial modality differences between radar and camera sensors pose challenges in fusing information. To address this problem, this paper presents RCBEVDet, a radar-camera fusion 3D object detection framework. Specifically, RCBEVDet is developed from an existing camera-based 3D object detector, supplemented by a specially designed radar feature extractor, RadarBEVNet, and a Cross-Attention Multi-layer Fusion (CAMF) module. Firstly, RadarBEVNet encodes sparse radar points into a dense bird's-eye-view (BEV) feature using a dual-stream radar backbone and a Radar Cross Section aware BEV encoder. Secondly, the CAMF module utilizes a deformable attention mechanism to align radar and camera BEV features and adopts channel and spatial fusion layers to fuse them. To further enhance RCBEVDet's capabilities, we introduce RCBEVDet++, which advances the CAMF through sparse fusion, supports query-based multi-view camera perception models, and adapts to a broader range of perception tasks. Extensive experiments on the nuScenes show that our method integrates seamlessly with existing camera-based 3D perception models and improves their performance across various perception tasks. Furthermore, our method achieves state-of-the-art radar-camera fusion results in 3D object detection, BEV semantic segmentation, and 3D multi-object tracking tasks. Notably, with ViT-L as the image backbone, RCBEVDet++ achieves 72.73 NDS and 67.34 mAP in 3D object detection without test-time augmentation or model ensembling.

Bridging the View Disparity Between Radar and Camera Features for Multi-Modal Fusion 3D Object Detection

Fusing LiDAR and Radar with Pillars Attention for 3D Object Detection

BEV-Radar: Bidirectional Radar-Camera Fusion for 3D Object Detection

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection

UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection

RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network

MVFusion: Multi-View 3D Object Detection with Semantic-aligned Radar and Camera Fusion

RCFusion: Fusing 4-D Radar and Camera with Bird's-Eye View Features for 3-D Object Detection.

Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving

Instance Fusion for Addressing Imbalanced Camera and Radar Data

Fusing Mmwave Radar with Camera for 3-D Detection in Autonomous Driving

RCBEVDet: Radar-camera Fusion in Bird’s Eye View for 3D Object Detection

Traffic Object Detection for Autonomous Driving Fusing LiDAR and Pseudo 4D-Radar under Bird’s-Eye-View

Two-Stage Feature Attention Fusion for Radar-Camera 3D Object Detection

SparseFusion3D: Sparse Sensor Fusion for 3D object detection by Radar and Camera in Environmental Perception

HVDetFusion: A Simple and Robust Camera-Radar Fusion Framework

Enhancing 3D object detection through multi-modal fusion for cooperative perception

RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion

Deep multi-scale and multi-modal fusion for 3D object detection

Temporal-enhanced Radar and Camera Fusion for Object Detection