Abstract:Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue. Since the computation complexity of ViT is quadratic with respect to the input sequence length, a mainstream paradigm for computation reduction is to reduce the number of tokens. Existing designs include structured spatial compression that uses a progressive shrinking pyramid to reduce the computations of large feature maps, and unstructured token pruning that dynamically drops redundant tokens. However, the limitation of existing token pruning lies in two folds: 1) the incomplete spatial structure caused by pruning is not compatible with structured spatial compression that is commonly used in modern deep-narrow transformers; 2) it usually requires a time-consuming pre-training procedure. To tackle the limitations and expand the applicable scenario of token pruning, we present Evo-ViT, a self-motivated slow-fast token evolution approach for vision transformers. Specifically, we conduct unstructured instance-wise token selection by taking advantage of the simple and effective global class attention that is native to vision transformers. Then, we propose to update the selected informative tokens and uninformative tokens with different computation paths, namely, slow-fast updating. Since slow-fast updating mechanism maintains the spatial structure and information flow, Evo-ViT can accelerate vanilla transformers of both flat and deep-narrow structures from the very beginning of the training process. Experimental results demonstrate that our method significantly reduces the computational cost of vision transformers while maintaining comparable performance on image classification. For example, our method accelerates DeiT-S by over 60% throughput while only sacrificing 0.4% top-1 accuracy on ImageNet-1K, outperforming current token pruning methods on both accuracy and efficiency.

Training-Free Acceleration of ViTs with Delayed Spatial Merging

Efficient Vision Transformer via Token Merger

Multi-Scale And Token Mergence: Make Your ViT More Efficient

Token Merging: Your ViT But Faster

GhostViT: Expediting Vision Transformers Via Cheap Operations

Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers

CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs

Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection

Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training

Make Your ViT-based Multi-view 3D Detectors Faster via Token Compression

Dynamic Token-Pass Transformers for Semantic Segmentation

Making Vision Transformers Efficient from A Token Sparsification View

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

vid-TLDR: Training Free Token merging for Light-weight Video Transformer

Accelerating Transformers with Spectrum-Preserving Token Merging

Token Cropr: Faster ViTs for Quite a Few Tasks

GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation

Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers

Merging Vision Transformers from Different Tasks and Domains

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations