Abstract:Point clouds obtained from 3D sensors are usually sparse. Existing methods mainly focus on upsampling sparse point clouds in a supervised manner by using dense ground truth point clouds. In this paper, we propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images. Specifically, we first propose a neighbor expansion unit (NEU) to upsample the sparse point clouds, where the local geometric structures of the sparse point clouds are exploited to learn weights for point interpolation. Then, we develop a differentiable point cloud rendering unit (DRU) as an end-to-end module in our network to render the point cloud into multi-view images. Finally, we formulate a shape-consistent loss and an image-consistent loss to train the network so that the shapes of the sparse and dense point clouds are as consistent as possible. Extensive results on the CAD and scanned datasets demonstrate that our method can achieve impressive results in a self-supervised manner. Code is available at <a class="link-external link-https" href="https://github.com/fpthink/SSPU-Net" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the problem of upsampling sparse point clouds. Point clouds obtained from 3D sensors are usually sparse, which makes it difficult to accurately capture the local geometric structure of 3D objects, thereby affecting the performance of subsequent point cloud processing tasks (such as 3D object segmentation, detection, and classification). Existing methods mainly use supervised learning to upsample sparse point clouds using dense real point clouds. However, in practical scenarios, obtaining dense real point clouds is very difficult. Therefore, this paper proposes a self-supervised point cloud upsampling network (SSPU-Net) that can generate dense point clouds without using real point clouds. ### Solution 1. **Neighborhood Expansion Unit (NEU)**: - A Neighborhood Expansion Unit (NEU) is proposed to learn weights by utilizing the local geometric structure of sparse point clouds to interpolate and generate new points. - Uses a 2D grid mechanism and self-attention mechanism to better fuse features. 2. **Differentiable Rendering Unit (DRU)**: - Developed a Differentiable Rendering Unit (DRU) that renders point clouds into multi-view images for end-to-end training. 3. **Shape-consistent Loss and Image-consistent Loss**: - Shape-consistent Loss: By minimizing the Earth Mover's Distance (EMD) between the input sparse point cloud and the downsampled dense point cloud, the shape of the dense point cloud is kept consistent with the sparse point cloud. - Image-consistent Loss: By minimizing the difference between the rendered images of the sparse point cloud and the dense point cloud, the local geometric structure of both is kept consistent. ### Experimental Results - **Quantitative Results**: Experimental results on the SSPU-DataSet and PU-GAN datasets show that SSPU-Net achieves comparable or even better results under self-supervised conditions compared to supervised methods. - **Visualization Results**: SSPU-Net can recover more details, such as the legs of a horse and the arms of a statue. - **Ablation Study**: Validated the effectiveness of the NEU module, shape-consistent loss, and image-consistent loss. - **Different Input Point Numbers**: SSPU-Net is robust to input point clouds of different densities. - **Generalization Ability**: Test results on the KITTI dataset show that SSPU-Net has good generalization ability. - **Downstream Tasks**: The upsampled point clouds can improve the performance of classification tasks. ### Main Contributions 1. **Proposed the Neighborhood Expansion Unit (NEU)**: Interpolates and generates new points by adaptively learning weights. 2. **Developed the Differentiable Rendering Unit (DRU)**: Renders point clouds into multi-view images for end-to-end training. 3. **Proposed Shape-consistent Loss and Image-consistent Loss**: Enables the point cloud upsampling network to be trained under self-supervised conditions. 4. **Achieved Impressive Results under Self-supervised Conditions**: Outperformed some supervised point cloud upsampling methods.

SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering

SPU-Net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction with Self-Projection Optimization

Self-Supervised Arbitrary-Scale Point Clouds Upsampling via Implicit Neural Representation

PU-Net: Point Cloud Upsampling Network

Self-Supervised Arbitrary-Scale Implicit Point Clouds Upsampling

SSPU-Net: A Structure Sensitive Point Cloud Upsampling Network with Multi-Scale Spatial Refinement

S3U-PVNet: Arbitrary-scale Point Cloud Upsampling Via Point-Voxel Network Based on Siamese Self-Supervised Learning

Point Cloud Upsampling via Cascaded Refinement Network

Density-imbalance-eased LiDAR Point Cloud Upsampling via Feature Consistency Learning

PU-SDF: Arbitrary-Scale Uniformly Upsampling Point Clouds Via Signed Distance Functions

Upu-Snet: Siamese Network for Unsupervised Point Cloud Upsampling Based on Spatial-Aware Transformers

Semantic Point Cloud Upsampling

Upsampling Autoencoder for Self-Supervised Point Cloud Learning

Point Cloud Upsampling Via Disentangled Refinement

PUI-Net: A Point Cloud Upsampling and Inpainting Network

Arbitrary-Scale Point Cloud Upsampling by Voxel-Based Network with Latent Geometric-Consistent Learning

APUNet: Attention-guided Upsampling Network for Sparse and Non-Uniform Point Cloud

Point Cloud Upsampling Algorithm: A Systematic Review

Data-driven Upsampling of Point Clouds

Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions

Parametric Surface Constrained Upsampler Network for Point Cloud