Abstract:The few-shot learning ability of vision transformers (ViTs) is rarely investigated though heavily desired. In this work, we empirically find that with the same few-shot learning frameworks, \eg~Meta-Baseline, replacing the widely used CNN feature extractor with a ViT model often severely impairs few-shot classification performance. Moreover, our empirical study shows that in the absence of inductive bias, ViTs often learn the low-qualified token dependencies under few-shot learning regime where only a few labeled training data are available, which largely contributes to the above performance degradation. To alleviate this issue, for the first time, we propose a simple yet effective few-shot training framework for ViTs, namely Self-promoted sUpervisioN (SUN). Specifically, besides the conventional global supervision for global semantic learning SUN further pretrains the ViT on the few-shot learning dataset and then uses it to generate individual location-specific supervision for guiding each patch token. This location-specific supervision tells the ViT which patch tokens are similar or dissimilar and thus accelerates token dependency learning. Moreover, it models the local semantics in each patch token to improve the object grounding and recognition capability which helps learn generalizable patterns. To improve the quality of location-specific supervision, we further propose two techniques:~1) background patch filtration to filtrate background patches out and assign them into an extra background class; and 2) spatial-consistent augmentation to introduce sufficient diversity for data augmentation while keeping the accuracy of the generated local supervisions. Experimental results show that SUN using ViTs significantly surpasses other few-shot learning frameworks with ViTs and is the first one that achieves higher performance than those CNN state-of-the-arts.

Reinforced Self-Supervised Training for Few-Shot Learning

Unsupervised Few-Shot Feature Learning Via Self-Supervised Training

Boosting Supervision with Self-Supervision for Few-shot Learning

Multi-task Self-supervised Few-Shot Detection.

Improving In-Context Few-Shot Learning via Self-Supervised Training

Few-Shot Classification with Multi-task Self-supervised Learning.

Task-Level Self-Supervision for Cross-Domain Few-Shot Learning

Self-Supervision Can Be a Good Few-Shot Learner

Three-Dimension Attention Mechanism and Self-Supervised Pretext Task for Augmenting Few-Shot Learning

Pareto Self-Supervised Training for Few-Shot Learning

Few-Shot Image Classification via Contrastive Self-Supervised Learning

Learning to Self-Train for Semi-Supervised Few-Shot Classification.

Unsupervised Few-shot Object Recognition by Self-Supervised Training

Self-Supervised Tuning for Few-Shot Segmentation

Self-supervised Knowledge Distillation for Few-shot Learning

A Theory of Self-Supervised Framework for Few-Shot Learning

SSL-DC: Improving Transductive Few-Shot Learning Via Self-Supervised Learning and Distribution Calibration.

Few-Shot Learning with Part Discovery and Augmentation from Unlabeled Images

Learning to Teach and Learn for Semi-Supervised Few-Shot Image Classification.

Self-Promoted Supervision for Few-Shot Transformer