Abstract:Masked Autoencoders (MAE) play a pivotal role in learning potent representations, delivering outstanding results across various 3D perception tasks essential for autonomous driving. In real-world driving scenarios, it's commonplace to deploy multiple sensors for comprehensive environment perception. Despite integrating multi-modal features from these sensors can produce rich and powerful features, there is a noticeable challenge in MAE methods addressing this integration due to the substantial disparity between the different modalities. This research delves into multi-modal Masked Autoencoders tailored for a unified representation space in autonomous driving, aiming to pioneer a more efficient fusion of two distinct modalities. To intricately marry the semantics inherent in images with the geometric intricacies of LiDAR point clouds, we propose UniM$^2$AE. This model stands as a potent yet straightforward, multi-modal self-supervised pre-training framework, mainly consisting of two designs. First, it projects the features from both modalities into a cohesive 3D volume space to intricately marry the bird's eye view (BEV) with the height dimension. The extension allows for a precise representation of objects and reduces information loss when aligning multi-modal features. Second, the Multi-modal 3D Interactive Module (MMIM) is invoked to facilitate the efficient inter-modal interaction during the interaction process. Extensive experiments conducted on the nuScenes Dataset attest to the efficacy of UniM$^2$AE, indicating enhancements in 3D object detection and BEV map segmentation by 1.2\% NDS and 6.5\% mIoU, respectively. The code is available at <a class="link-external link-https" href="https://github.com/hollow-503/UniM2AE" rel="external noopener nofollow">this https URL</a>.

Poster Abstract: Representation Learning from Multimodal Sensor Data with Maximally Correlated Autoencoders

Cross-Modality Feature Learning Via Convolutional Autoencoder

CrossMAE: Cross Modality Masked Autoencoders for Region-Aware Audio-Visual Pretraining

Multimodal Learning of Social Image Representation by Exploiting Social Relations

UniM^2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition

Learning Classifiers from Synthetic Data Using a Multichannel Autoencoder.

UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

Deep Coupling Autoencoder for Fault Diagnosis with Multimodal Sensory Data

Multi-context Autoencoders for Multivariate Medical Signals Based on Deep Convolutional Neural Networks

Common Representation Learning Using Step-based Correlation Multi-Modal CNN

MULTI-LABEL IMAGE RECOGNITION WITH JOINT CLASS-AWARE MAP DISENTANGLING AND LABEL CORRELATION EMBEDDING

Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

A New Modal Autoencoder for Functionally Independent Feature Extraction

Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing

Towards Transferable Multi-modal Perception Representation Learning for Autonomy: NeRF-Supervised Masked AutoEncoder

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection

Representation Learning With Dual Autoencoder for Multi-Label Classification

Unsupervised Multimodal Language Representations using Convolutional Autoencoders

A Markov Random Field Multi-Modal Variational AutoEncoder

Fus-MAE: A cross-attention-based data fusion approach for Masked Autoencoders in remote sensing