Abstract:Three dimensional (3D) object detection using the camera and light detection and ranging (LiDAR) fusion model has received much attention for meeting the environmental sensing requirement in autonomous vehicles. Due to the intrinsic data difference and the uneven distribution of LiDAR data, integrating the camera and LiDAR to solve the problem of the uneven spatial distribution of LiDAR data is still challenging. This paper proposes a novel 3D detector, which decorates the point cloud voxels with the semantic features at multi-scale and fuses the multi-source features by cross-attention to improve the detection performance. Specifically, the multi-modal fusion aims to maintain the cross-modal consistency by using the frustum model to construct the correspondence between point voxels and image pixels at multi-scale. To fully exploit the geometric constraints of voxels, we developed a adaptive sampling radius module to dynamically select the sampling radius of the voxel set abstraction module based on softmax. Moreover, we propose a multi-source fusion module which utilizes the cross-attention and takes the raw point cloud's spatial distribution as the clue to fuse the features of point clouds, bird's eye view information and the aggregated voxels to obtain the multi-scale and multi-source fusion features. Finally, region proposal network is adopted to generate and refine the 3D bbox and class prediction based on the fusion feature and bird's eye view feature. Extensive experiments are conducted on the two publicly available 3D object datasets proposed by Karlsruhe Institute of technology and Toyota Technological Institute (KITTI) and the nuScenes. The proposed model reaches 86.27% mean Average Precision (mAP) for car on KITTI and obtains 6.56% gains than PointPainting. Moreover, our model reaches 70.70% mAP on nuScens and improves 2.61% than Transfusion. Furthermore, comprehensive ablation experiments are conducted to validate the effects and contribution of the different components of the proposed model.

MSIT-Det: Multi-Scale Feature Aggregation with Iterative Transformer Networks for 3D Object Detection

MT-SSD: Single-Stage 3D Object Detector Based on Magnification Transformation

Dynamic graph transformer for 3D object detection

HCT-Det: a Hybrid CNN-transformer Architecture for 3D Object Detection from Point Clouds

Sparse Embedded Convolution Based Dual Feature Aggregation 3D Object Detection Network

TraIL-Det: Transformation-Invariant Local Feature Networks for 3D LiDAR Object Detection with Unsupervised Pre-Training

TSSTDet: Transformation-Based 3-D Object Detection via a Spatial Shape Transformer

MS23D: A 3D object detection method using multi-scale semantic feature points to construct 3D feature layer

Improving 3D Object Detection with Channel-wise Transformer

Li3DeTr: A LiDAR based 3D Detection Transformer

DS-Trans: A 3D Object Detection Method Based on a Deformable Spatiotemporal Transformer for Autonomous Vehicles

MSL3D: 3D object detection from monocular, stereo and point cloud for autonomous driving

Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection

Spatial Information Enhancement with Multi-Scale Feature Aggregation for Long-Range Object and Small Reflective Area Object Detection from Point Cloud

LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention

Cascade fusion of multi-modal and multi-source feature fusion by the attention for three-dimensional object detection

CT3D++: Improving 3D Object Detection with Keypoint-induced Channel-wise Transformer

MSPV3D: Multi-Scale Point-Voxels 3D Object Detection Net

AGO-Net: Association-Guided 3D Point Cloud Object Detection Network

PTA-Det: Point Transformer Associating Point Cloud and Image for 3D Object Detection

SparseDet: A Simple and Effective Framework for Fully Sparse LiDAR-based 3D Object Detection