Abstract:Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{<a class="link-external link-https" href="https://github.com/ZhichengHuang/CMAE" rel="external noopener nofollow">this https URL</a>}.

MultiMAE: Multi-modal Multi-task Masked Autoencoders

CrossMAE: Cross Modality Masked Autoencoders for Region-Aware Audio-Visual Pretraining

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection

Multimodal Masked Autoencoders Learn Transferable Representations

CL-MAE: Curriculum-Learned Masked Autoencoders

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization

ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders

AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders

Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts

UniM^2AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

Masked Autoencoders Are Scalable Vision Learners

UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving

Contrastive Masked Autoencoders are Stronger Vision Learners

MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning

Pseudo Labelling for Enhanced Masked Autoencoders

CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets

Efficient Masked Autoencoders with Self-Consistency

Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders