Abstract:Instance segmentation on 3D point clouds (3DIS) is a longstanding challenge in computer vision, where state-of-the-art methods are mainly based on full supervision. As annotating ground truth dense instance masks is tedious and expensive, solving 3DIS with weak supervision has become more practical. In this paper, we propose GaPro, a new instance segmentation for 3D point clouds using axis-aligned 3D bounding box supervision. Our two-step approach involves generating pseudo labels from box annotations and training a 3DIS network with the resulting labels. Additionally, we employ the self-training strategy to improve the performance of our method further. We devise an effective Gaussian Process to generate pseudo instance masks from the bounding boxes and resolve ambiguities when they overlap, resulting in pseudo instance masks with their uncertainty values. Our experiments show that GaPro outperforms previous weakly supervised 3D instance segmentation methods and has competitive performance compared to state-of-the-art fully supervised ones. Furthermore, we demonstrate the robustness of our approach, where we can adapt various state-of-the-art fully supervised methods to the weak supervision task by using our pseudo labels for training. The source code and trained models are available at <a class="link-external link-https" href="https://github.com/VinAIResearch/GaPro" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of weakly - supervised learning in **3D point - cloud - based instance segmentation (3DIS)**, especially using **axis - aligned 3D bounding boxes as supervision signals**. Specifically, the paper proposes a new method - **GaPro** to generate pseudo - labels during the training process using only axis - aligned 3D bounding boxes, thereby achieving 3D point - cloud instance segmentation. #### Main challenges: 1. **High annotation cost**: Fully - supervised 3D point - cloud instance segmentation requires manual annotation of instance masks for each point, which is very time - consuming and expensive. 2. **Ambiguity under weak supervision**: Axis - aligned 3D bounding boxes can only roughly represent the scope of an object and cannot capture the specific shape or geometric structure of the object, resulting in ambiguity in the assignment between points and objects, especially in areas where multiple bounding boxes overlap. 3. **Limitations of existing methods**: Previous weakly - supervised methods (such as Box2Mask) perform poorly in dealing with point - object assignment ambiguity, resulting in performance far below that of fully - supervised methods. #### Solutions: - **GaPro** solves the ambiguity problem in the overlapping areas of bounding boxes by introducing the **Gaussian Process (GP)** to generate pseudo - labels and their uncertainty values. - **Two - step method**: 1. **Generate pseudo - labels**: Use the Gaussian process to generate pseudo - labels and uncertainty values from the bounding box annotations. 2. **Train the 3DIS network**: Train the 3DIS network using the generated pseudo - labels and optimize it by combining KL - divergence loss and mask loss. - **Self - training strategy**: Further improve the model performance by iteratively improving the quality of pseudo - labels. #### Experimental results: - GaPro significantly outperforms other weakly - supervised methods on the ScanNetV2 and S3DIS datasets and in some cases approaches the performance of fully - supervised methods. - The effectiveness of each component, including the Gaussian process, KL - divergence loss, and self - training strategy, is verified through ablation experiments. In conclusion, this paper successfully addresses the key challenges of weakly - supervised learning in 3D point - cloud instance segmentation by proposing the GaPro method, providing an efficient and robust solution.

GaPro: Box-Supervised 3D Point Cloud Instance Segmentation Using Gaussian Processes as Pseudo Labelers