Abstract:Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at <a class="link-external link-https" href="https://github.com/youweiliang/evit" rel="external noopener nofollow">this https URL</a>

Token Selection is a Simple Booster for Vision Transformers

Vision Transformer with Super Token Sampling

Predicting Token Impact Towards Efficient Vision Transformer

DualToken-ViT: Position-aware Efficient Vision Transformer with Dual Token Fusion

DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.

Constituent Attention for Vision Transformers

No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling

Making Vision Transformers Efficient from A Token Sparsification View

Robustifying Token Attention for Vision Transformers

Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention

An Attention-Based Token Pruning Method for Vision Transformers

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

All Tokens Matter: Token Labeling for Training Better Vision Transformers

LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

Efficient Video Transformers with Spatial-Temporal Token Selection

Transformer with token attention and attribute prediction for image captioning

Refiner: Refining Self-attention for Vision Transformers

Token Labeling: Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet.

Multi-Scale And Token Mergence: Make Your ViT More Efficient