Abstract:In this paper, we present a new approach for model acceleration by exploiting spatial sparsity in visual data. We observe that the final prediction in vision Transformers is only based on a subset of the most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input to accelerate vision Transformers. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. While the framework is inspired by our observation of the sparse attention in vision Transformers, we find the idea of adaptive and asymmetric computation can be a general solution for accelerating various architectures. We extend our method to hierarchical models including CNNs and hierarchical vision Transformers as well as more complex dense prediction tasks that require structured feature maps by formulating a more generic dynamic spatial sparsification framework with progressive sparsification and asymmetric computation for different spatial locations. By applying lightweight fast paths to less informative features and using more expressive slow paths to more important locations, we can maintain the structure of feature maps while significantly reducing the overall computations. Extensive experiments demonstrate the effectiveness of our framework on various modern architectures and different visual recognition tasks. Our results clearly demonstrate that dynamic spatial sparsification offers a new and more effective dimension for model acceleration. Code is available at <a class="link-external link-https" href="https://github.com/raoyongming/DynamicViT" rel="external noopener nofollow">this https URL</a>

ShaderTransformer: Predicting Shader Quality Via One-shot Embedding for Fast Simplification

Runtime Shader Simplification Via Instant Search in Reduced Optimization Space.

Automatic shader simplification using surface signal approximation

Multirate Shading with Piecewise Interpolatory Approximation

Appearance-Driven Automatic 3D Model Simplification

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Identifying shader sub-patterns for GPU performance tuning and architecture design

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

Neural Mesh Simplification

AFD-former: A Hybrid Transformer with Asymmetric Flow Division for Synthesized View Quality Enhancement

Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understanding

Accelerating Vision Diffusion Transformers with Skip Branches

A Compact Transformer for Adaptive Style Transfer

Scene Diffusion: Text-driven Scene Image Synthesis Conditioning on a Single 3D Model

Depth Estimation with Simplified Transformer

Automatic Mesh and Shader Level of Detail

Learning from Shader Program Traces

Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

ExtraNet: Real-time Extrapolated Rendering for Low-latency Temporal Supersampling

Efficient Shapley Values Calculation for Transformer Explainability

EHA-Transformer: Efficient and Haze-Adaptive Transformer for Single Image Dehazing