Abstract:Masked Autoencoders (MAE) play a pivotal role in learning potent representations, delivering outstanding results across various 3D perception tasks essential for autonomous driving. In real-world driving scenarios, it's commonplace to deploy multiple sensors for comprehensive environment perception. Despite integrating multi-modal features from these sensors can produce rich and powerful features, there is a noticeable challenge in MAE methods addressing this integration due to the substantial disparity between the different modalities. This research delves into multi-modal Masked Autoencoders tailored for a unified representation space in autonomous driving, aiming to pioneer a more efficient fusion of two distinct modalities. To intricately marry the semantics inherent in images with the geometric intricacies of LiDAR point clouds, we propose UniM$^2$AE. This model stands as a potent yet straightforward, multi-modal self-supervised pre-training framework, mainly consisting of two designs. First, it projects the features from both modalities into a cohesive 3D volume space to intricately marry the bird's eye view (BEV) with the height dimension. The extension allows for a precise representation of objects and reduces information loss when aligning multi-modal features. Second, the Multi-modal 3D Interactive Module (MMIM) is invoked to facilitate the efficient inter-modal interaction during the interaction process. Extensive experiments conducted on the nuScenes Dataset attest to the efficacy of UniM$^2$AE, indicating enhancements in 3D object detection and BEV map segmentation by 1.2\% NDS and 6.5\% mIoU, respectively. The code is available at <a class="link-external link-https" href="https://github.com/hollow-503/UniM2AE" rel="external noopener nofollow">this https URL</a>.

Masked Autoencoders for Egocentric Video Understanding @ Ego4D Challenge 2022

Masked Autoencoders for Point Cloud Self-supervised Learning.

EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation

InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges

PCIE_LAM Solution for Ego4D Looking At Me Challenge

Egocentric Video Task Translation @ Ego4D Challenge 2022

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

Audiovisual Masked Autoencoders

Masked Autoencoders for Generic Event Boundary Detection CVPR'2022 Kinetics-GEBD Challenge

Masked Autoencoders As Spatiotemporal Learners

GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition

Helping Hands: An Object-Aware Ego-Centric Video Recognition Model

Technical Report for Ego4D Long Term Action Anticipation Challenge 2023

MV2MAE: Multi-View Video Masked Autoencoders

UniM^2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

EgoEnv: Human-centric environment representations from egocentric video

Domain-Guided Masked Autoencoders for Unique Player Identification

UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

A Simple Transformer-Based Model for Ego4D Natural Language Queries Challenge

Adversarially Masked Video Consistency for Unsupervised Domain Adaptation