Abstract:Self-supervised learning has achieved great success in both natural language processing and 2D vision, where masked modeling is a quite popular pre-training scheme. However, extending masking to 3D point cloud understanding that combines local and global features poses a new challenge. In our work, we present Point-LGMask, a novel method to embed both local and global contexts with multi-ratio masking, which is quite effective for self-supervised feature learning of point clouds but is unfortunately ignored by existing pre-training works. Specifically, to avoid fitting to a fixed masking ratio, we first propose multi-ratio masking, which prompts the encoder to fully explore representative features thanks to tasks of different difficulties. Next, to encourage the embedding of both local and global features, we formulate a compound loss, which consists of (i) a global representation contrastive loss to encourage the cluster assignments of the masked point clouds to be consistent to that of the completed input, and (ii) a local point cloud prediction loss to encourage accurate prediction of masked points. Equipped with our Point-LGMask, we show that our learned representations transfer well to various downstream tasks, including few-shot classification, shape classification, object part segmentation, as well as real-world scene-based 3D object detection and 3D semantic segmentation. Particularly, our model largely advances existing pre-training methods on the difficult few-shot classification task using the real-captured ScanObjectNN dataset by surpassing over 4% to the second-best method. Also, our Point-LGMask achieves 0.4% $AP_{25}$ and 0.8% $AP_{50}$ gains on 3D object detection task over the second-best method. 0.4% mAcc and 0.5% mIoU. Codes have been released at https://github.com/TangYuan96/Point-LGMask .

GeoMask3D: Geometrically Informed Mask Selection for Self-Supervised Point Cloud Learning in 3D

Masked Autoencoders for Point Cloud Self-supervised Learning.

Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder

GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training

M^3CS: Multi-Target Masked Point Modeling with Learnable Codebook and Siamese Decoders

Masked Autoencoders in 3D Point Cloud Representation Learning

Point‐AGM : Attention Guided Masked Auto‐Encoder for Joint Self‐supervised Learning on Point Clouds

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

Point Cloud Domain Adaptation Via Masked Local 3D Structure Prediction

Triple Point Masking

CMAE-3D: Contrastive Masked AutoEncoders for Self-Supervised 3D Object Detection

Point-LGMask: Local and Global Contexts Embedding for Point Cloud Pre-training with Multi-Ratio Masking

LR-MAE: Locate While Reconstructing with Masked Autoencoders for Point Cloud Self-supervised Learning

Masked Generative Extractor for Synergistic Representation and 3D Generation of Point Clouds

Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training

PatchMixing Masked Autoencoders for 3D Point Cloud Self-Supervised Learning

PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders

Inter-Modal Masked Autoencoder for Self-Supervised Learning on Point Clouds