Abstract:With the emergence of large-scale models trained on diverse datasets, in-context learning has emerged as a promising paradigm for multitasking, notably in natural language processing and image processing. However, its application in 3D point cloud tasks remains largely unexplored. In this work, we introduce Point-In-Context (PIC), a novel framework for 3D point cloud understanding via in-context learning. We address the technical challenge of effectively extending masked point modeling to 3D point clouds by introducing a Joint Sampling module and proposing a vanilla version of PIC called Point-In-Context-Generalist (PIC-G). PIC-G is designed as a generalist model for various 3D point cloud tasks, with inputs and outputs modeled as coordinates. In this paradigm, the challenging segmentation task is achieved by assigning label points with XYZ coordinates for each category; the final prediction is then chosen based on the label point closest to the predictions. To break the limitation by the fixed label-coordinate assignment, which has poor generalization upon novel classes, we propose two novel training strategies, In-Context Labeling and In-Context Enhancing, forming an extended version of PIC named Point-In-Context-Segmenter (PIC-S), targeting improving dynamic context labeling and model training. By utilizing dynamic in-context labels and extra in-context pairs, PIC-S achieves enhanced performance and generalization capability in and across part segmentation datasets. PIC is a general framework so that other tasks or datasets can be seamlessly introduced into our PIC through a unified data format. We conduct extensive experiments to validate the versatility and adaptability of our proposed methods in handling a wide range of tasks and segmenting multi-datasets. Our PIC-S is capable of generalizing unseen datasets and performing novel part segmentation by customizing prompts.

ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

Instance-Aware Embedding for Point Cloud Instance Segmentation

DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

3D Semantic Segmentation Using Deep Learning for Large-Scale Indoor Point Cloud

Joint Semantic-Instance Segmentation of 3D Point Clouds: Instance Separation and Semantic Fusion.

3D-BEVIS: Bird's-Eye-View Instance Segmentation

3DCFS: Fast and Robust Joint 3D Semantic-Instance Segmentation via Coupled Feature Selection

PointInst3D: Segmenting 3D Instances by Points

Unsupervised Joint 3D Object Model Learning and 6D Pose Estimation for Depth-Based Instance Segmentation.

Investigate Indistinguishable Points in Semantic Segmentation of 3D Point Cloud

Automated Semantic Segmentation of Indoor Point Clouds from Close-Range Images with Three-Dimensional Deep Learning

Associatively Segmenting Instances and Semantics in Point Clouds

TPV-IGKD: Image-Guided Knowledge Distillation for 3D Semantic Segmentation with Tri-Plane-View

Neighborhood co-occurrence modeling in 3D point cloud segmentation

OccuSeg: Occupancy-Aware 3D Instance Segmentation

Semantic Labeling and Instance Segmentation of 3D Point Clouds Using Patch Context Analysis and Multiscale Processing

Point-In-Context: Understanding Point Cloud via In-Context Learning

Deep Projective 3D Semantic Segmentation

Dynamic Convolution for 3D Point Cloud Instance Segmentation