Abstract:Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP, which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to achieve 3D point cloud part segmentation with a small amount of labeled data (zero-shot or few-shot). Specifically, the paper proposes a method called PartSLIP, which leverages pre-trained image-language models (such as GLIP) to achieve semantic and instance segmentation of 3D point clouds. This method aims to reduce the dependency on large-scale 3D labeled data, thereby improving the model's generalization ability on new categories. ### Background 3D part segmentation is very important in computer vision and robotics, but traditional supervised learning methods require large 3D datasets with fine-grained part annotations, which are very costly to collect. Therefore, how to achieve effective 3D part segmentation with a small amount of labeled data has become an important research problem. ### Solution The paper proposes the PartSLIP method, which mainly includes the following steps: 1. **Multi-view Rendering**: Render the 3D point cloud into 2D images from multiple predefined camera views. 2. **GLIP Model Detection**: Input these 2D images and text prompts into the pre-trained GLIP model, which outputs 2D bounding boxes for each view. 3. **2D to 3D Conversion**: Through a novel 3D voting and grouping module, fuse the multi-view 2D bounding boxes into 3D instance segmentation results. 4. **Prompt Tuning**: Use a small amount of labeled 3D shapes for prompt tuning, enabling the GLIP model to quickly adapt to specific part definitions. 5. **Multi-view Feature Aggregation**: Enhance the GLIP model's global understanding of the input 3D shape through a multi-view feature aggregation module. ### Experimental Results The paper conducts extensive evaluations on the PartNet and PartNet-Mobility datasets, showing that the PartSLIP method achieves excellent performance in both zero-shot and few-shot settings. Particularly in the few-shot setting, PartSLIP not only significantly outperforms existing few-shot methods but also competes with fully supervised methods. ### Main Contributions 1. **Innovative 3D Part Segmentation Method**: Utilizes pre-trained image-language models to achieve zero-shot and few-shot 3D part segmentation. 2. **3D Voting and Grouping Module**: Effectively converts multi-view 2D bounding boxes into 3D semantic and instance segmentation. 3. **Prompt Tuning and Multi-view Feature Aggregation**: Improves the detection performance of the GLIP model. 4. **New Benchmark Dataset**: Proposes the PartNetE dataset, which facilitates future research in low-shot and text-driven 3D part segmentation. Through these contributions, the paper provides a new and efficient method for the 3D part segmentation field, capable of achieving high-performance segmentation results with a small amount of labeled data.

PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models

PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimation

Leveraging Large-Scale Pretrained Vision Foundation Models for Label-Efficient 3D Point Cloud Segmentation

CAM/CAD Point Cloud Part Segmentation via Few-Shot Learning

ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation

PartSeg: Few-shot Part Segmentation via Part-aware Prompt Learning

PointCLIP: Point Cloud Understanding by CLIP

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation

3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation

SAMPart3D: Segment Any Part in 3D Objects

Generalized Few-Shot Point Cloud Segmentation Via Geometric Words

A Closer Look at Few-Shot 3D Point Cloud Classification

SegPoint: Segment Any Point Cloud via Large Language Model

CLIP^2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

A Simple Framework of Few-Shot Learning Using Sparse Annotations for Semantic Segmentation of 3-D Point Clouds

MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition

PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation

PointSeg: Real-Time Semantic Segmentation Based on 3D LiDAR Point Cloud

PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning