Abstract:Self-supervised learning has achieved great success in both natural language processing and 2D vision, where masked modeling is a quite popular pre-training scheme. However, extending masking to 3D point cloud understanding that combines local and global features poses a new challenge. In our work, we present Point-LGMask, a novel method to embed both local and global contexts with multi-ratio masking, which is quite effective for self-supervised feature learning of point clouds but is unfortunately ignored by existing pre-training works. Specifically, to avoid fitting to a fixed masking ratio, we first propose multi-ratio masking, which prompts the encoder to fully explore representative features thanks to tasks of different difficulties. Next, to encourage the embedding of both local and global features, we formulate a compound loss, which consists of (i) a global representation contrastive loss to encourage the cluster assignments of the masked point clouds to be consistent to that of the completed input, and (ii) a local point cloud prediction loss to encourage accurate prediction of masked points. Equipped with our Point-LGMask, we show that our learned representations transfer well to various downstream tasks, including few-shot classification, shape classification, object part segmentation, as well as real-world scene-based 3D object detection and 3D semantic segmentation. Particularly, our model largely advances existing pre-training methods on the difficult few-shot classification task using the real-captured ScanObjectNN dataset by surpassing over 4% to the second-best method. Also, our Point-LGMask achieves 0.4% $AP_{25}$ and 0.8% $AP_{50}$ gains on 3D object detection task over the second-best method. 0.4% mAcc and 0.5% mIoU. Codes have been released at https://github.com/TangYuan96/Point-LGMask .

MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation

PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation

Spherical Mask: Coarse-to-Fine 3D Point Cloud Instance Segmentation with Spherical Representation

MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation

DualGroup for 3D Instance and Panoptic Segmentation

Scalable SoftGroup for 3D Instance Segmentation on Point Clouds

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

An Efficient Ungrouped Mask Method with Two Learnable Parameters for 3D Object Detection

DynaMask: Dynamic Mask Selection for Instance Segmentation

OccuSeg: Occupancy-Aware 3D Instance Segmentation

Point-LGMask: Local and Global Contexts Embedding for Point Cloud Pre-training with Multi-Ratio Masking

Divide and Conquer: 3D Point Cloud Instance Segmentation With Point-Wise Binarization

Hierarchical Aggregation for 3D Instance Segmentation

SPGroup3D: Superpoint Grouping Network for Indoor 3D Object Detection

MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features

End-to-End 3D Point Cloud Instance Segmentation Without Detection

PointInst3D: Segmenting 3D Instances by Points

BorderPointsMask: One-stage Instance Segmentation with Boundary Points Representation.

RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features

Part segmentation method of point cloud considering optimal allocation and optimal mask based on deep learning