SegPoint: Segment Any Point Cloud via Large Language Model

Shuting He,Henghui Ding,Xudong Jiang,Bihan Wen

2024-07-19

Abstract:Despite significant progress in 3D point cloud segmentation, existing methods primarily address specific tasks and depend on explicit instructions to identify targets, lacking the capability to infer and understand implicit user intentions in a unified framework. In this work, we propose a model, called SegPoint, that leverages the reasoning capabilities of a multi-modal Large Language Model (LLM) to produce point-wise segmentation masks across a diverse range of tasks: 1) 3D instruction segmentation, 2) 3D referring segmentation, 3) 3D semantic segmentation, and 4) 3D open-vocabulary semantic segmentation. To advance 3D instruction research, we introduce a new benchmark, Instruct3D, designed to evaluate segmentation performance from complex and implicit instructional texts, featuring 2,565 point cloud-instruction pairs. Our experimental results demonstrate that SegPoint achieves competitive performance on established benchmarks such as ScanRefer for referring segmentation and ScanNet for semantic segmentation, while delivering outstanding outcomes on the Instruct3D dataset. To our knowledge, SegPoint is the first model to address these varied segmentation tasks within a single framework, achieving satisfactory performance.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address multiple tasks in 3D point cloud segmentation and proposes a unified framework to handle these tasks. Specifically, existing methods mainly target specific tasks and rely on explicit instructions to identify target objects, lacking the ability to infer and understand implicit user intentions within a unified framework. To solve this problem, the authors propose the SegPoint model, which leverages the reasoning capabilities of multimodal large language models (LLM) to generate point-level segmentation masks across different 3D point cloud segmentation tasks, including:

1. **3D Instruction Segmentation**: Segmentation through complex and implicit instruction texts.
2. **3D Referenced Segmentation**: Segmentation of target instances based on explicit textual descriptions.
3. **3D Semantic Segmentation**: Assigning specific semantic categories to each point in the point cloud.
4. **3D Open Vocabulary Semantic Segmentation**: Identifying and segmenting new objects beyond known categories.

Additionally, to advance research on 3D segmentation based on complex instructions, the authors introduce a new benchmark dataset, Instruct3D, containing 2,565 point cloud-instruction pairs, to evaluate the model's segmentation performance under complex and implicit instructions. Experimental results show that SegPoint not only performs well on existing benchmark datasets such as ScanRefer and ScanNet but also achieves significant results on the newly proposed Instruct3D dataset. This is the first model to address multiple 3D segmentation tasks within a single framework.