ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

Le Xue,Mingfei Gao,Chen Xing,Roberto Martín-Martín,Jiajun Wu,Caiming Xiong,Ran Xu,Juan Carlos Niebles,Silvio Savarese

2023-06-13

Abstract:The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models are released at <a class="link-external link-https" href="https://github.com/salesforce/ULIP" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to improve the performance of 3D visual understanding tasks through joint learning of multi - modal information (language, image, and point cloud). Specifically, current 3D models have limited performance when dealing with datasets with less labeled data and limited categories. To solve this problem, the authors proposed ULIP (Learning a Unified Representation of Language, Images, and Point Clouds), aiming to enhance the understanding ability of 3D point clouds by pre - training with large - scale image - text pairs.

### Specific Background of the Problem

1. **Limitations of 3D Data**:
- Current 3D datasets are small in scale. For example, ShapeNet55 contains only about 52.5K samples and 55 categories.
- In contrast, 2D datasets such as ImageNet contain millions of images and cover thousands of categories.
- The high cost of collecting and labeling 3D data limits the generalization ability and practical applications of 3D models.

2. **Advantages of Multi - modal Learning**:
- Existing research shows that using knowledge from different modalities can significantly help the conceptual understanding of the original modality. For example, CLIP significantly improves the visual concept recognition ability through large - scale image - text pair pre - training and achieves zero - shot classification.
- However, research on multi - modal learning involving 3D modalities is still insufficient, especially in terms of how to use this multi - modal information to improve 3D recognition tasks.

### The Core Idea of ULIP

To overcome the problem of insufficient 3D data, ULIP proposes a new framework, which is achieved in the following ways:

1. **Create Triplet Data**:
- Generate triplets containing images, text descriptions, and point clouds from ShapeNet55.
- Use a small number of automatically generated triplets for pre - training.

2. **Align Multi - modal Feature Spaces**:
- Utilize the image - text feature space already learned by pre - trained vision - language models (such as CLIP).
- Align the features of the 3D point cloud encoder to this common feature space.

3. **Contrastive Learning**:
- Use a contrastive loss function to align the feature representations of images, text, and point clouds, ensuring that they are aligned in the same feature space.

### Main Contributions

1. **Significantly Improve 3D Recognition Performance**:
- On standard 3D classification and zero - shot 3D classification tasks, ULIP significantly improves the performance of multiple 3D backbone networks.
- For example, on the ModelNet40 and ScanObjectNN datasets, ULIP respectively improves the classification accuracy of PointMLP and PointBERT.

2. **Potential for Cross - modal Applications**:
- Aligning the feature spaces of the three modalities makes more cross - domain downstream tasks possible, such as zero - shot 3D classification and image - to - point - cloud retrieval.

### Summary

This paper successfully solves the problem of insufficient 3D data by introducing the ULIP framework and shows the great potential of multi - modal learning in 3D visual understanding. The experimental results show that ULIP can not only significantly improve the performance of existing 3D models but also provide new ideas for future cross - modal applications.