PV-SSD: A Multi-Modal Point Cloud Feature Fusion Method for Projection Features and Variable Receptive Field Voxel Features

Yongxin Shao,Aihong Tan,Zhetao Sun,Enhui Zheng,Tianhong Yan,Peng Liao
2024-04-13
Abstract:LiDAR-based 3D object detection and classification is crucial for autonomous driving. However, real-time inference from extremely sparse 3D data is a formidable challenge. To address this problem, a typical class of approaches transforms the point cloud cast into a regular data representation (voxels or projection maps). Then, it performs feature extraction with convolutional neural networks. However, such methods often result in a certain degree of information loss due to down-sampling or over-compression of feature information. This paper proposes a multi-modal point cloud feature fusion method for projection features and variable receptive field voxel features (PV-SSD) based on projection and variable voxelization to solve the information loss problem. We design a two-branch feature extraction structure with a 2D convolutional neural network to extract the point cloud's projection features in bird's-eye view to focus on the correlation between local features. A voxel feature extraction branch is used to extract local fine-grained features. Meanwhile, we propose a voxel feature extraction method with variable sensory fields to reduce the information loss of voxel branches due to downsampling. It avoids missing critical point information by selecting more useful feature points based on feature point weights for the detection task. In addition, we propose a multi-modal feature fusion module for point clouds. To validate the effectiveness of our method, we tested it on the KITTI dataset and ONCE dataset.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve The paper aims to address the problem of 3D object detection based on LiDAR in autonomous driving scenarios. Specifically, the paper focuses on the sparsity issue of point cloud data in autonomous driving scenarios. Due to the sparsity of point cloud data, converting these data into conventional data representations (such as voxels or projections) usually leads to information loss. This information loss can severely affect detection accuracy, especially for objects with fewer reflection points (such as cyclists). To solve this problem, the paper proposes a multi-modal point cloud 3D object detector based on projection features and voxel features (PV-SSD). This detector consists of two branches: 1. **Voxel Branch**: Used to extract fine-grained local features. 2. **Projection Branch**: Used to extract projection features from the bird's-eye view and focus on the correlation of local features in the voxel branch. By inputting voxel features into the projection branch, it can compensate for the information loss in the projection branch while focusing on the correlation between adjacent local features in the voxel features. To achieve comprehensive fusion of voxel features and projection features, the paper proposes a multi-modal feature fusion module (MSSFA). Additionally, to further reduce the loss of key features caused by downsampling, the paper also proposes a voxel feature extraction method based on the importance of feature points (VR-VFE). ### Experimental Validation To verify the effectiveness of the proposed method, the researchers conducted tests on the KITTI dataset and the ONCE dataset. The experimental results show that the method achieves significant accuracy improvements in detecting objects with fewer reflection points (such as cyclists).