Abstract:Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at <a class="link-external link-https" href="https://github.com/youweiliang/evit" rel="external noopener nofollow">this https URL</a>

Efficient Vision Transformer via Token Merger

Making Vision Transformers Efficient from A Token Sparsification View

Multi-Scale And Token Mergence: Make Your ViT More Efficient

No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling

Token Fusion: Bridging the Gap between Token Pruning and Token Merging

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.

Efficient Visual Transformer by Learnable Token Merging

GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation

Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer

Dynamic Token-Pass Transformers for Semantic Segmentation

PPT: Token Pruning and Pooling for Efficient Vision Transformers

Learned Thresholds Token Merging and Pruning for Vision Transformers

Training-Free Acceleration of ViTs with Delayed Spatial Merging

Super Vision Transformer

Token Merging: Your ViT But Faster

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers

Token Cropr: Faster ViTs for Quite a Few Tasks

Efficient Online Inference of Vision Transformers by Training-Free Tokenization

TPC-ViT: Token Propagation Controller for Efficient Vision Transformer