Abstract:In this paper, we present a new approach for model acceleration by exploiting spatial sparsity in visual data. We observe that the final prediction in vision Transformers is only based on a subset of the most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input to accelerate vision Transformers. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. While the framework is inspired by our observation of the sparse attention in vision Transformers, we find the idea of adaptive and asymmetric computation can be a general solution for accelerating various architectures. We extend our method to hierarchical models including CNNs and hierarchical vision Transformers as well as more complex dense prediction tasks that require structured feature maps by formulating a more generic dynamic spatial sparsification framework with progressive sparsification and asymmetric computation for different spatial locations. By applying lightweight fast paths to less informative features and using more expressive slow paths to more important locations, we can maintain the structure of feature maps while significantly reducing the overall computations. Extensive experiments demonstrate the effectiveness of our framework on various modern architectures and different visual recognition tasks. Our results clearly demonstrate that dynamic spatial sparsification offers a new and more effective dimension for model acceleration. Code is available at <a class="link-external link-https" href="https://github.com/raoyongming/DynamicViT" rel="external noopener nofollow">this https URL</a>

ASCEND: Accurate yet Efficient End-to-End Stochastic Computing Acceleration of Vision Transformer

Not Your Father’s Stochastic Computing (SC)! Efficient Yet Accurate End-to-End SC Accelerator Design

Accurate yet Efficient Stochastic Computing Neural Acceleration with High Precision Residual Fusion

Efficient yet Accurate End-to-End SC Accelerator Design

GhostViT: Expediting Vision Transformers Via Cheap Operations

NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict Aware Supernet Training

HSViT: A Hardware and Software Collaborative Design for Vision Transformer Via Multi-level Compression

SPViT: Enabling Faster Vision Transformers Via Latency-Aware Soft Token Pruning

ReViT: Vision Transformer Accelerator with Reconfigurable Semantic-aware Differential Attention

Boost Vision Transformer with GPU-Friendly Sparsity and Quantization

Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios

Making Vision Transformers Efficient from A Token Sparsification View

Vision Transformer with Sparse Scan Prior

Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

Auto-scaling Vision Transformers without Training

Super Vision Transformer

ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention

UCC: A Unified Cascade Compression Framework for Vision Transformer Models

ViTA: A Vision Transformer Inference Accelerator for Edge Applications

ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration