Abstract:Vision and language foundation models (VLMs) have showcased impressive capabilities in 2D scene understanding. However, their latent potential in elevating the understanding of 3D autonomous driving scenes remains untapped. In this paper, we propose VLM2Scene, which exploits the potential of VLMs to enhance 3D self-supervised representation learning through our proposed image-text-LiDAR contrastive learning strategy. Specifically, in the realm of autonomous driving scenes, the inherent sparsity of LiDAR point clouds poses a notable challenge for point-level contrastive learning methods. This method often grapples with limitations tied to a restricted receptive field and the presence of noisy points. To tackle this challenge, our approach emphasizes region-level learning, leveraging regional masks without semantics derived from the vision foundation model. This approach capitalizes on valuable contextual information to enhance the learning of point cloud representations. First, we introduce Region Caption Prompts to generate fine-grained language descriptions for the corresponding regions, utilizing the language foundation model. These region prompts then facilitate the establishment of positive and negative text-point pairs within the contrastive loss framework. Second, we propose a Region Semantic Concordance Regularization, which involves a semantic-filtered region learning and a region semantic assignment strategy. The former aims to filter the false negative samples based on the semantic distance, and the latter mitigates potential inaccuracies in pixel semantics, thereby enhancing overall semantic consistency. Extensive experiments on representative autonomous driving datasets demonstrate that our self-supervised method significantly outperforms other counterparts. Codes are available at https://github.com/gbliao/VLM2Scene.

ContrastMotion: Self-supervised Scene Motion Learning for Large-Scale LiDAR Point Clouds

Self-supervised Visual-LiDAR Odometry with Flip Consistency

Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos

Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation

SegContrast: 3D Point Cloud Feature Representation Learning Through Self-Supervised Segment Discrimination

VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding

Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds

MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps

MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps

Self-Supervised Class-Agnostic Motion Prediction with Spatial and Temporal Consistency Regularizations

Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals

BE-STI: Spatial-Temporal Integrated Network for Class-agnostic Motion Prediction with Bidirectional Enhancement

BEVScope: Enhancing Self-Supervised Depth Estimation Leveraging Bird's-Eye-View in Dynamic Scenarios

Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving

Cohere3D: Exploiting Temporal Coherence for Unsupervised Representation Learning of Vision-based Autonomous Driving

Unsupervised 3D Point Cloud Representation Learning by Triangle Constrained Contrast for Autonomous Driving.

GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding

MotionBEV: Attention-Aware Online LiDAR Moving Object Segmentation with Bird's Eye View based Appearance and Motion Features

Motion Sensitive Contrastive Learning for Self-supervised Video Representation

Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences

Directed Mix Contrast for Lidar Point Cloud Segmentation