PI-Net: an End-to-End Deep Neural Network for Bidirectionally and Directly Fusing Point Clouds with Images

Qi Wang,Jian Chen,Jianqiang Deng,Xinfang Zhang

DOI: https://doi.org/10.1109/lra.2021.3114429

2021-01-01

Abstract:We present a novel network, PI-Net, for the fusion between point clouds and images in this letter. Most existing fusion methods project point clouds into pseudo images and then fuse the pseudo and RGB images with 2D CNNs. To get rid of structuring the pseudo images as the preprocessing, we propose a new fusion module, PI-fusion module. It can directly fuse points with pixels or grids which refer to the cells in the feature maps extracted from RGB images. Specifically, every point is aligned with a certain pixel/grid based on the pre-calibrated camera-LiDAR external parameters. The features aggregated from the points and pixels/grids are transferred to each other, which achieves the bidirectional fusion of the information. Then, we construct the PI-Net's fusion backbone by plugging the PI-fusion modules between a certain 2D CNN and a point-based network to fuse the features at every scale. The detection and the segmentation head networks further utilize the fused information to complete various tasks. The PI-Net unifies the point-cloud-to-image and image-to-point-cloud fusion into one network and achieves state-of-the-art results on the various tasks, such as the road detection, 3D object detection, and point cloud segmentation tasks on the KITTI benchmark.