Abstract:Vision Transformers (ViTs) outperforms convolutional neural networks (CNNs) in several vision tasks with its global modeling capabilities. However, ViT lacks the inductive bias inherent to convolution making it require a large amount of data for training. This results in ViT not performing as well as CNNs on small datasets like medicine and science. We experimentally found that masked autoencoders (MAE) can make the transformer focus more on the image itself, thus alleviating the data-hungry issue of ViT to some extent. Yet the current MAE model is too complex resulting in over-fitting problems on small datasets. This leads to a gap between MAEs trained on small datasets and advanced CNNs models still. Therefore, we investigated how to reduce the decoder complexity in MAE and found a more suitable architectural configuration for it with small datasets. Besides, we additionally designed a location prediction task and a contrastive learning task to introduce localization and invariance characteristics for MAE. Our contrastive learning task not only enables the model to learn high-level visual information but also allows the training of MAE's class token. This is something that most MAE improvement efforts do not consider. Extensive experiments have shown that our method shows state-of-the-art performance on standard small datasets as well as medical datasets with few samples compared to the current popular masked image modeling (MIM) and vision transformers for small <a class="link-external link-http" href="http://datasets.The" rel="external noopener nofollow">this http URL</a> code and models are available at <a class="link-external link-https" href="https://github.com/Talented-Q/SDMAE" rel="external noopener nofollow">this https URL</a>.

Denoising masked autoencoders help robust classification

Denoising Masked Autoencoders Help Robust Classification.

Masked Autoencoders for Point Cloud Self-supervised Learning.

A Label Noise Robust Stacked Auto-Encoder Algorithm for Inaccurate Supervised Classification Problems

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

Masked Image Training for Generalizable Deep Image Denoising

SdAE: Self-distillated Masked Autoencoder

Masked Autoencoders Are Scalable Vision Learners

Masked autoencoders are effective solution to transformer data-hungry

DEMAE: Diffusion-Enhanced Masked Autoencoder for Hyperspectral Image Classification With Few Labeled Samples

Masked Autoencoders Enable Efficient Knowledge Distillers

Masked Autoencoders are Efficient Class Incremental Learners

Denoising Auto-Encoders Toward Robust Unsupervised Feature Representation.

LoMAE: Low-level Vision Masked Autoencoders for Low-dose CT Denoising

Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification

LoMAE: Simple Streamlined Low-level Masked Autoencoders for Robust, Generalized, and Interpretable Low-dose CT Denoising

Point-DAE: Denoising Autoencoders for Self-supervised Point Cloud Learning

VideoMAC: Video Masked Autoencoders Meet ConvNets

Efficient Masked Autoencoders with Self-Consistency

Teaching Masked Autoencoder With Strong Augmentations