Abstract:Large vision models have achieved great success in computer vision recently, e.g., CLIP for large-scale image-text contrastive learning. They have prominent potential in representation learning and show strong transfer ability in various downstream tasks. However, directly training a larger CLIP model from scratch is difficult because of the enormous training cost, unstable training, and difficulty in collecting a large amount of training data. In this work, we aim to scale the sizes of CLIP models and extend their strong capabilities with self-supervised representation learning. We introduce masked channel modeling (MCM), a new self-supervised learning framework that randomly masks the input feature maps extracted by a CLIP model and reconstructs the missing features. Unlike masked image modeling (MIM) which takes raw pixels as the input and output, MCM performs masked modeling at a high-dimensional semantic space by masking random channels of the visual features and reconstructing the corrupted channels. We show that channel maps are a great fit for masked modeling, as the visual features are semantically structured across channels. We demonstrate that our method can easily scale up the CLIP model at a low training cost, and extend its capabilities on zero-shot learning, few-shot learning, and end-to-end fine-tuning. Based on CLIP ViT-L, MCM improves the zero-shot image classification accuracy by 0.5% averaged over 8 benchmarks. With a few samples, e.g., 1-shot or 2-shot, MCM achieves significant improvements when adapting to 11 image classification benchmarks. In addition, MCM shows strong performance when end-to-end fine-tuned on different downstream tasks, e.g., improving CLIP ViT-B by 0.9% top-1 accuracy on ImageNet-1K classification and 2.5% mIoU on ADE20K semantic segmentation.

Toward High Quality Facial Representation Learning

Towards Mask-robust Face Recognition.

General Facial Representation Learning in a Visual-Linguistic Manner

MIMIC: Mask Image Pre-training with Mix Contrastive Fine-tuning for Facial Expression Recognition

Adept: Annotation-denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining

Self-Attention and MLP Auxiliary Convolution for Face Anti-Spoofing

MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining

PrefAce: Face-Centric Pretraining with Self-Structure Aware Distillation

Blindfold Attention: Novel Mask Strategy for Facial Expression Recognition

FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning

Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation

Self-Supervised Facial Representation Learning with Facial Region Awareness

Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations

Masked Contrastive Representation Learning

Multi-Level Contrastive Learning for Dense Prediction Task

Pose-disentangled Contrastive Learning for Self-supervised Facial Representation

Progressive Self-supervised Representation Learning for 3D Facial Expression Recognition

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Masked Contrastive Pre-Training for Efficient Video-Text Retrieval

Masked Channel Modeling for Bootstrapping Visual Pre-training

Detecting Masked Faces in the Wild with LLE-CNNs