Abstract:Three dimensional (3D) object detection using the camera and light detection and ranging (LiDAR) fusion model has received much attention for meeting the environmental sensing requirement in autonomous vehicles. Due to the intrinsic data difference and the uneven distribution of LiDAR data, integrating the camera and LiDAR to solve the problem of the uneven spatial distribution of LiDAR data is still challenging. This paper proposes a novel 3D detector, which decorates the point cloud voxels with the semantic features at multi-scale and fuses the multi-source features by cross-attention to improve the detection performance. Specifically, the multi-modal fusion aims to maintain the cross-modal consistency by using the frustum model to construct the correspondence between point voxels and image pixels at multi-scale. To fully exploit the geometric constraints of voxels, we developed a adaptive sampling radius module to dynamically select the sampling radius of the voxel set abstraction module based on softmax. Moreover, we propose a multi-source fusion module which utilizes the cross-attention and takes the raw point cloud's spatial distribution as the clue to fuse the features of point clouds, bird's eye view information and the aggregated voxels to obtain the multi-scale and multi-source fusion features. Finally, region proposal network is adopted to generate and refine the 3D bbox and class prediction based on the fusion feature and bird's eye view feature. Extensive experiments are conducted on the two publicly available 3D object datasets proposed by Karlsruhe Institute of technology and Toyota Technological Institute (KITTI) and the nuScenes. The proposed model reaches 86.27% mean Average Precision (mAP) for car on KITTI and obtains 6.56% gains than PointPainting. Moreover, our model reaches 70.70% mAP on nuScens and improves 2.61% than Transfusion. Furthermore, comprehensive ablation experiments are conducted to validate the effects and contribution of the different components of the proposed model.

Robust-FusionNet: Deep Multimodal Sensor Fusion for 3-D Object Detection Under Severe Weather Conditions

3D object detection algorithm based on multi-sensor segmental fusion of frustum association for autonomous driving

An Adaptive Multimodal Fusion 3D Object Detection Algorithm for Unmanned Systems in Adverse Weather

Three-Dimensional Object Detection Network Based on Multi-Layer and Multi-Modal Fusion

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

3DMMF: 3D Object Detection Network Based on Multi-Layer and Multi-Modal Fusion

Robust Target Recognition and Tracking of Self-Driving Cars With Radar and Camera Information Fusion Under Severe Weather Conditions

Enhancing 3D object detection through multi-modal fusion for cooperative perception

TransFusion: Multi-Modal Robust Fusion for 3D Object Detection in Foggy Weather Based on Spatial Vision Transformer

3D Object Detection Algorithm in Adverse Weather Conditions Based on LiDAR-Radar Fusion

ContextualFusion: Context-Based Multi-Sensor Fusion for 3D Object Detection in Adverse Operating Conditions

All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark

Pay "Attention" to Adverse Weather: Weather-aware Attention-based Object Detection

L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection

ACF-Net: Asymmetric Cascade Fusion for 3D Detection with LiDAR Point Clouds and Images

Frustum FusionNet: Amodal 3D Object Detection with Multi-Modal Feature Fusion

MMFusion: A Generalized Multi-Modal Fusion Detection Framework

Multi-sensor Fusion 3D Object Detection Based on Channel Attention

Cascade fusion of multi-modal and multi-source feature fusion by the attention for three-dimensional object detection

Multi-Modal and Multi-Scale Fusion 3D Object Detection of 4D Radar and LiDAR for Autonomous Driving

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning