Abstract:Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{<a class="link-external link-https" href="https://github.com/ZhichengHuang/CMAE" rel="external noopener nofollow">this https URL</a>}.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to combine the advantages of Contrastive Learning (CL) and Masked Image Modeling (MIM) in self-supervised learning to enhance the effectiveness of visual representation learning. Specifically, while MIM methods have shown promising results in various visual tasks, the representations they learn have limited discriminative ability, indicating significant room for improvement. To overcome this limitation, the paper proposes Contrastive Masked Autoencoders (CMAE), a novel self-supervised pre-training method designed to learn more comprehensive and robust visual representations by cleverly unifying CL and MIM. CMAE introduces two new components—pixel shifting (for generating reasonable positive sample views) and a feature decoder (for supplementing the features of contrastive pairs)—to make contrastive learning compatible with MIM. These designs enable CMAE to effectively improve representation quality and achieve state-of-the-art performance on multiple benchmarks, such as ImageNet classification, semantic segmentation, and object detection tasks. The main contributions of the paper include: 1. Proposing the CMAE method, exploring how to use contrastive learning to enhance MIM's learned representations, allowing it to retain local context-sensitive features while also modeling instance discrimination between different images. 2. Introducing a feature decoder to supplement masked features and a weak spatial shifting augmentation method to generate reasonable contrastive views, both of which effectively improve the quality of encoder features. 3. Experimental results show that CMAE significantly enhances MIM's learned representations, achieving new state-of-the-art levels, such as a 0.7% absolute improvement on the ImageNet-1K classification validation set and a 1.8% mIoU improvement on the ADE20K semantic segmentation validation dataset. Overall, through innovative design, the paper successfully combines contrastive learning with MIM, providing a new solution for visual representation learning.

Contrastive Masked Autoencoders are Stronger Vision Learners

Efficient Masked Autoencoders with Self-Consistency

Learning with Unmasked Tokens Drives Stronger Vision Learners

Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget

A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

CL-MAE: Curriculum-Learned Masked Autoencoders

Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning

Understanding Masked Autoencoders From a Local Contrastive Perspective

Masked Autoencoders are Efficient Class Incremental Learners

SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners

Contrastive Audio-Visual Masked Autoencoder

MAE-MACD: the Masked Adversarial Contrastive Distillation Algorithm Grounded in Masked Autoencoders

Context Autoencoder for Self-Supervised Representation Learning

Masked Contrastive Representation Learning

Teaching Masked Autoencoder With Strong Augmentations

Progressively Compressed Auto-Encoder for Self-supervised Representation Learning.

Stare at What You See: Masked Image Modeling Without Reconstruction

Masked Image Modeling with Denoising Contrast

CMAE-V: Contrastive Masked Autoencoders for Video Action Recognition

Masked Autoencoders Are Scalable Vision Learners