Abstract:Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{<a class="link-external link-https" href="https://github.com/ZhichengHuang/CMAE" rel="external noopener nofollow">this https URL</a>}.

CD-MAE: Contrastive Dual-Masked Autoencoder Pre-Training Model for PCB CT Image Element Segmentation

Efficient Pretraining Model based on Multi-Scale Local Visual Field Feature Reconstruction for PCB CT Image Element Segmentation

Tissue-Contrastive Semi-Masked Autoencoders for Segmentation Pretraining on Chest CT

PCMask: A Dual-Branch Self-supervised Medical Image Segmentation Method Using Pixel-Level Contrastive Learning and Masked Image Modeling.

MAE-MACD: the Masked Adversarial Contrastive Distillation Algorithm Grounded in Masked Autoencoders

Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification

PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

An active learning method based on result quality evaluation for printed circuit board computed tomography image segmentation

Contrastive Masked Autoencoders are Stronger Vision Learners

CMAE-3D: Contrastive Masked AutoEncoders for Self-Supervised 3D Object Detection

A Masked Autoencoder -Based Approach for Defect Classification in Semiconductor Manufacturing

Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts

Learning the Heterogeneous Representation of Brain's Structure from Serial SEM Images Using a Masked Autoencoder

Enhancing Semi-Supervised Medical Image Segmentation with Bidirectional Copy-Paste and Masked Image Reconstruction

Self-Supervised Pre-Training Based on Contrastive Complementary Masking for Semi-Supervised Cardiac Image Segmentation

SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders

Teaching Masked Autoencoder With Strong Augmentations

Transformer-Based Masked Autoencoder with Contrastive Loss for Hyperspectral Image Classification.