LidarCLIP or: How I Learned to Talk to Point Clouds

Georg Hess,Adam Tonderski,Christoffer Petersson,Kalle Åström,Lennart Svensson

DOI: https://doi.org/10.48550/arXiv.2212.06858

2023-05-02

Abstract:Research connecting text and images has recently seen several breakthroughs, with models like CLIP, DALL-E 2, and Stable Diffusion. However, the connection between text and other visual modalities, such as lidar data, has received less attention, prohibited by the lack of text-lidar datasets. In this work, we propose LidarCLIP, a mapping from automotive point clouds to a pre-existing CLIP embedding space. Using image-lidar pairs, we supervise a point cloud encoder with the image CLIP embeddings, effectively relating text and lidar data with the image domain as an intermediary. We show the effectiveness of LidarCLIP by demonstrating that lidar-based retrieval is generally on par with image-based retrieval, but with complementary strengths and weaknesses. By combining image and lidar features, we improve upon both single-modality methods and enable a targeted search for challenging detection scenarios under adverse sensor conditions. We also explore zero-shot classification and show that LidarCLIP outperforms existing attempts to use CLIP for point clouds by a large margin. Finally, we leverage our compatibility with CLIP to explore a range of applications, such as point cloud captioning and lidar-to-image generation, without any additional training. Code and pre-trained models are available at <a class="link-external link-https" href="https://github.com/atonderski/lidarclip" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

The paper aims to address the issue of connecting natural language processing (NLP) with LiDAR point cloud data. Despite significant progress in linking images and text in recent years, such as models like CLIP, DALL·E 2, and Stable Diffusion, there has been relatively little research on connecting text with other visual modalities like LiDAR data. The main reason is the lack of large-scale text-LiDAR datasets.

### Specific Problems

1. **Lack of large-scale text-LiDAR datasets**: Existing research mainly focuses on images and text, while LiDAR data, due to its specificity, is difficult to obtain large-scale annotated data.
2. **Limitations of existing methods**: Current attempts to connect NLP with point clouds are usually limited to single applications or designed for synthetic data, failing to fully utilize large-scale raw autonomous driving data.
3. **Challenges of multimodal fusion**: How to combine LiDAR data with image data to improve detection and retrieval performance in complex scenes.

### Solution

To overcome these issues, the authors propose LidarCLIP, a method to embed LiDAR point clouds into the existing CLIP embedding space. Specifically:

- **Dataset utilization**: Utilizing large-scale autonomous driving datasets (such as ONCE), which contain a large number of image-LiDAR pairs.
- **Training method**: By supervising a LiDAR encoder to match the features it generates with those generated by a frozen CLIP image encoder, thereby transferring the rich semantic understanding from the image domain to the LiDAR point cloud domain.
- **Multimodal fusion**: By combining image and LiDAR features, LidarCLIP can perform well in various application scenarios, including zero-shot classification, scene retrieval, point cloud description generation, and LiDAR-to-image generation.

### Main Contributions

1. **Proposing LidarCLIP**: A new method to embed LiDAR point clouds into the existing CLIP embedding space.
2. **Effectiveness validation**: LidarCLIP outperforms existing CLIP-based methods in zero-shot classification and retrieval tasks.
3. **Complementarity**: LidarCLIP is complementary to the CLIP teacher model, even outperforming CLIP in certain retrieval categories. By combining the two methods, performance is further improved, especially in retrieving key scenes under complex perception conditions.
4. **Multiple applications**: Demonstrating the potential of LidarCLIP in applications such as point cloud description generation and LiDAR-to-image generation without additional training.

### Conclusion

LidarCLIP connects LiDAR point cloud data with the CLIP embedding space, not only addressing the lack of large-scale text-LiDAR datasets but also providing new ideas for multimodal fusion, which is expected to play an important role in autonomous driving and other fields.