Abstract: Masked AutoEncoder (MAE) has recently led the trends of visual self-supervision area by an elegant asymmetric encoder-decoder design, which significantly optimizes both the pre-training efficiency and fine-tuning accuracy. Notably, the success of the asymmetric structure relies on the "global" property of Vanilla Vision Transformer (ViT), whose self-attention mechanism reasons over arbitrary subset of discrete image patches. However, it is still unclear how the advanced Pyramid-based ViTs (e.g., PVT, Swin) can be adopted in MAE pre-training as they commonly introduce operators within "local" windows, making it difficult to handle the random sequence of partial vision tokens. In this paper, we propose Uniform Masking (UM), successfully enabling MAE pre-training for Pyramid-based ViTs with locality (termed "UM-MAE" for short). Specifically, UM includes a Uniform Sampling (US) that strictly samples $1$ random patch from each $2 \times 2$ grid, and a Secondary Masking (SM) which randomly masks a portion of (usually $25\%$) the already sampled regions as learnable tokens. US preserves equivalent elements across multiple non-overlapped local windows, resulting in the smooth support for popular Pyramid-based ViTs; whilst SM is designed for better transferable visual representations since US reduces the difficulty of pixel recovery pre-task that hinders the semantic learning. We demonstrate that UM-MAE significantly improves the pre-training efficiency (e.g., it speeds up and reduces the GPU memory by $\sim 2\times$) of Pyramid-based ViTs, but maintains the competitive fine-tuning performance across downstream tasks. For example using HTC++ detector, the pre-trained Swin-Large backbone self-supervised under UM-MAE only in ImageNet-1K can even outperform the one supervised in ImageNet-22K. The codes are available at https://github.com/implus/UM-MAE.

VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding

VLMAE: Vision-Language Masked Autoencoder

CrossMAE: Cross Modality Masked Autoencoders for Region-Aware Audio-Visual Pretraining

Masked autoencoders are effective solution to transformer data-hungry

Domain-oriented Language Pre-training with Adaptive Hybrid Masking and Optimal Transport Alignment

Domain-oriented Language Modeling with Adaptive Hybrid Masking and Optimal Transport Alignment

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders

E-VarM: Enhanced Variational Word Masks to Improve the Interpretability of Text Classification Models.

Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality

VAE-based Domain Adaptation for Speaker Verification.

SparseMAE: Sparse Training Meets Masked Autoencoders.

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Meta conditional variational auto-encoder for domain generalization

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

LoMAE: Low-level Vision Masked Autoencoders for Low-dose CT Denoising

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation

RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models

MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder