Abstract:Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue. Since the computation complexity of ViT is quadratic with respect to the input sequence length, a mainstream paradigm for computation reduction is to reduce the number of tokens. Existing designs include structured spatial compression that uses a progressive shrinking pyramid to reduce the computations of large feature maps, and unstructured token pruning that dynamically drops redundant tokens. However, the limitation of existing token pruning lies in two folds: 1) the incomplete spatial structure caused by pruning is not compatible with structured spatial compression that is commonly used in modern deep-narrow transformers; 2) it usually requires a time-consuming pre-training procedure. To tackle the limitations and expand the applicable scenario of token pruning, we present Evo-ViT, a self-motivated slow-fast token evolution approach for vision transformers. Specifically, we conduct unstructured instance-wise token selection by taking advantage of the simple and effective global class attention that is native to vision transformers. Then, we propose to update the selected informative tokens and uninformative tokens with different computation paths, namely, slow-fast updating. Since slow-fast updating mechanism maintains the spatial structure and information flow, Evo-ViT can accelerate vanilla transformers of both flat and deep-narrow structures from the very beginning of the training process. Experimental results demonstrate that our method significantly reduces the computational cost of vision transformers while maintaining comparable performance on image classification. For example, our method accelerates DeiT-S by over 60% throughput while only sacrificing 0.4% top-1 accuracy on ImageNet-1K, outperforming current token pruning methods on both accuracy and efficiency.

Automated Progressive Learning for Efficient Training of Vision Transformers

Efficient Training of Large Vision Models via Advanced Automated Progressive Learning

Auto-scaling Vision Transformers without Training

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

GhostViT: Expediting Vision Transformers Via Cheap Operations

Vision Transformer with Progressive Sampling

Effective Vision Transformer Training: A Data-Centric Perspective

Super Vision Transformer

Training a Vision Transformer from scratch in less than 24 hours with 1 GPU

AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Improving Vision Transformers for Incremental Learning

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Boosting Vanilla Lightweight Vision Transformers Via Re-parameterization

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training.

Denoising Vision Transformers

Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning

Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training

Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets

No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer