Abstract:It has been well recognized that fusing the complementary information from depth-aware LiDAR point clouds and semantic-rich stereo images would benefit 3D object detection. Nevertheless, it is non-trivial to explore the inherently unnatural interaction between sparse 3D points and dense 2D pixels. To ease this difficulty, the recent approaches generally project the 3D points onto the 2D image plane to sample the image data and then aggregate the data at the points. However, these approaches often suffer from the mismatch between the resolution of point clouds and RGB images, leading to sub-optimal performance. Specifically, taking the sparse points as the multi-modal data aggregation locations causes severe information loss for high-resolution images, which in turn undermines the effectiveness of multi-sensor fusion. In this paper, we present VPFNet a new architecture that cleverly aligns and aggregates the point cloud and image data at the virtual points. Particularly, with their density lying between that of the 3D points and 2D pixels, the virtual points can nicely bridge the resolution gap between the two sensors, and thus preserve more information for processing. Moreover, we also investigate the data augmentation techniques that can be applied to both point clouds and RGB images, as the data augmentation has made non-negligible contribution towards 3D object detectors to date. We have conducted extensive experiments on KITTI dataset, and have observed good performance compared to the state-of-the-art methods. Remarkably, our VPFNet achieves 83.21 moderate $AP_{3D}$ and 91.86 moderate $AP_{BEV}$ on the KITTI test set. The network design also takes computation efficiency into consideration we can achieve a FPS of 15 on a single NVIDIA RTX 2080Ti GPU.

EppNet: Enhanced Pseudo and Point Cloud Fusion for 3D Object Detection

SSF: Sparse Point Cloud Object Detection Based on Self-Adaptive Voxel Encoding and Focal-Sparse Convolution

EPNet++: Cascade Bi-Directional Fusion for Multi-Modal 3D Object Detection

FS-Net: LiDAR-Camera Fusion With Matched Scale for 3D Object Detection in Autonomous Driving

PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module

AEPF: Attention-Enabled Point Fusion for 3D Object Detection

Point-Voxel Fusion for 3D Object Detection

VirPNet: A Multimodal Virtual Point Generation Network for 3D Object Detection

FFPA-Net: Efficient Feature Fusion with Projection Awareness for 3D Object Detection

VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion

Adaptive Learning Point Cloud and Image Diversity Feature Fusion Network for 3D Object Detection

Three-Dimensional Point Cloud Object Detection Based on Feature Fusion and Enhancement

SP-Net: A Sparse Convolution and Point-Encoding Enhanced Network for 3D Object Detection in LiDAR Point Clouds.

Murf-Net: Multi-Receptive Field Pillars For 3d Object Detection From Point Cloud

PPF-Det: Point-Pixel Fusion for Multi-Modal 3D Object Detection

VPFNet: Voxel-Pixel Fusion Network for Multi-class 3D Object Detection

3D Object Detector: A Multiscale Region Proposal Network Based on Autonomous Driving.

EFNet: enhancing feature information for 3D object detection in LiDAR point clouds

FuseNet: 3D Object Detection Network with Fused Information for Lidar Point Clouds.

Multifeature Fusion-Based Object Detection for Intelligent Transportation Systems

VoPiFNet: Voxel-Pixel Fusion Network for Multi-Class 3D Object Detection