P2M2-Net: Part-Aware Prompt-Guided Multimodal Point Cloud Completion

Linlian Jiang,Pan Chen,Ye Wang,Tieru Wu,Rui Ma

DOI: https://doi.org/10.48550/arXiv.2312.17611

2023-12-29

Abstract:Inferring missing regions from severely occluded point clouds is highly challenging. Especially for 3D shapes with rich geometry and structure details, inherent ambiguities of the unknown parts are existing. Existing approaches either learn a one-to-one mapping in a supervised manner or train a generative model to synthesize the missing points for the completion of 3D point cloud shapes. These methods, however, lack the controllability for the completion process and the results are either deterministic or exhibiting uncontrolled diversity. Inspired by the prompt-driven data generation and editing, we propose a novel prompt-guided point cloud completion framework, coined P2M2-Net, to enable more controllable and more diverse shape completion. Given an input partial point cloud and a text prompt describing the part-aware information such as semantics and structure of the missing region, our Transformer-based completion network can efficiently fuse the multimodal features and generate diverse results following the prompt guidance. We train the P2M2-Net on a new large-scale PartNet-Prompt dataset and conduct extensive experiments on two challenging shape completion benchmarks. Quantitative and qualitative results show the efficacy of incorporating prompts for more controllable part-aware point cloud completion and generation. Code and data are available at <a class="link-external link-https" href="https://github.com/JLU-ICL/P2M2-Net" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the paper attempts to solve

The paper aims to solve the difficult problem of inferring missing regions from severely occluded point clouds. Especially for 3D shapes with rich geometric structure details, there is inherent ambiguity in the unknown parts. Existing methods either learn one - to - one mappings through supervised learning or synthesize missing points through generative models to complete 3D point cloud shapes. However, these methods lack controllability over the completion process, and the results are either deterministic or show uncontrolled diversity.

To improve the controllability and diversity of point cloud completion, the authors propose a new prompt - based point cloud completion framework - P2M2 - Net. This framework utilizes text prompts to describe the semantic and structural information of missing regions, efficiently fuses multi - modal features through the Transformer network, and generates diverse results according to the prompts.

### Main contributions

1. **Constructing the PartNet - Prompt dataset**: This is a large - scale dataset that contains part - level text prompt annotations. Through paired cross - modal data (the semantics and structure of shape parts), it supports various applications, such as part - aware point cloud completion and generation, fine - grained shape understanding and retrieval.

2. **Proposing the P2M2 - Net framework**: This is a new part - aware prompt - guided framework that can achieve more controllable point cloud completion. Through contrastive pre - training and a multi - modal feature encoder, it better aligns and fuses cross - modal features. After training is completed, using different text prompts as guidance, P2M2 - Net can generate diverse results from a single input.

3. **Experimental verification**: Extensive experiments were carried out on two challenging PartNet benchmarks. The quantitative and qualitative comparison results show the effectiveness of guided completion combined with prompts. In addition, an ablation study of the modules was carried out to verify the effectiveness of each module.

### Solutions

1. **Dataset construction**: Manually label parts of representative shapes in the PartNet dataset to generate short - text prompts that describe the geometric, structural, and semantic information of the parts.

2. **Contrastive pre - training**: A contrastive pre - training module was designed, which includes a point cloud encoder and a text - prompt encoder, as well as two MLP projection heads to map the features to a unified embedding space. The InfoNCE loss function is used to align the point cloud and text - prompt features.

3. **Multi - modal Transformer**: The PoinTr network was extended to be able to handle multi - modal features. The multi - modal feature encoder fuses features through self - attention and cross - attention layers to generate a coarse point cloud. The multi - modal query generator generates query proxies for querying relevant features. The multi - modal point cloud decoder predicts detailed local shapes and finally generates a complete shape.

### Experimental results

1. **Quantitative comparison**: On the PartNet and PartNet - Scan benchmarks, comparisons were made with representative and state - of - the - art point cloud completion methods. The results show that P2M2 - Net outperforms the baseline methods in most cases.

2. **Qualitative comparison**: The qualitative results of different methods on the two benchmarks were shown. P2M2 - Net can generate shapes that meet expectations, especially performing better in some complex situations.

3. **Multi - modal completion evaluation**: Using different text prompts as inputs, P2M2 - Net can generate diverse completion results. Quantitative and qualitative comparisons show that P2M2 - Net is superior to other methods in both quality and diversity.

### Ablation study

An ablation study was carried out to evaluate the effectiveness of the contrastive pre - training and attention - based multi - modal feature fusion modules. The results verify the importance of each module.

In conclusion, P2M2 - Net achieves more controllable and diverse point cloud completion by introducing text prompts, solving the problems of insufficient controllability and diversity in the completion process of existing methods.