Abstract:Current perception models in autonomous driving heavily rely on large-scale labelled 3D data, which is both costly and time-consuming to annotate. This work proposes a solution to reduce the dependence on labelled 3D training data by leveraging pre-training on large-scale unlabeled outdoor LiDAR point clouds using masked autoencoders (MAE). While existing masked point autoencoding methods mainly focus on small-scale indoor point clouds or pillar-based large-scale outdoor LiDAR data, our approach introduces a new self-supervised masked occupancy pre-training method called Occupancy-MAE, specifically designed for voxel-based large-scale outdoor LiDAR point clouds. Occupancy-MAE takes advantage of the gradually sparse voxel occupancy structure of outdoor LiDAR point clouds and incorporates a range-aware random masking strategy and a pretext task of occupancy prediction. By randomly masking voxels based on their distance to the LiDAR and predicting the masked occupancy structure of the entire 3D surrounding scene, Occupancy-MAE encourages the extraction of high-level semantic information to reconstruct the masked voxel using only a small number of visible voxels. Extensive experiments demonstrate the effectiveness of Occupancy-MAE across several downstream tasks. For 3D object detection, Occupancy-MAE reduces the labelled data required for car detection on the KITTI dataset by half and improves small object detection by approximately 2% in AP on the Waymo dataset. For 3D semantic segmentation, Occupancy-MAE outperforms training from scratch by around 2% in mIoU. For multi-object tracking, Occupancy-MAE enhances training from scratch by approximately 1% in terms of AMOTA and AMOTP. Codes are publicly available at <a class="link-external link-https" href="https://github.com/chaytonmin/Occupancy-MAE" rel="external noopener nofollow">this https URL</a>.

BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios

Masked Autoencoders for Point Cloud Self-supervised Learning.

BEV-MAE: Bird's Eye View Masked Autoencoders for Outdoor Point Cloud Pre-training

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

Masked Autoencoder for Pre-Training on 3D Point Cloud Object Detection

Masked Autoencoder for Self-Supervised Pre-training on Lidar Point Clouds

Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoders

A Streamlined Framework for Bev-Based 3d Object Detection with Prior Masking

Point Cloud Self-supervised Learning via 3D to Multi-view Masked Autoencoder

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection

Inter-Modal Masked Autoencoder for Self-Supervised Learning on Point Clouds

LR-MAE: Locate While Reconstructing with Masked Autoencoders for Point Cloud Self-supervised Learning

M-BEV: Masked BEV Perception for Robust Autonomous Driving

Masked Autoencoders in 3D Point Cloud Representation Learning

Learning Shared RGB-D Fields: Unified Self-supervised Pre-training for Label-efficient LiDAR-Camera 3D Perception

PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds

UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

UniM^2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning

Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training