Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

Yongming Rao,Zuyan Liu,Wenliang Zhao,Jie Zhou,Jiwen Lu

2023-06-02

Abstract:In this paper, we present a new approach for model acceleration by exploiting spatial sparsity in visual data. We observe that the final prediction in vision Transformers is only based on a subset of the most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input to accelerate vision Transformers. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. While the framework is inspired by our observation of the sparse attention in vision Transformers, we find the idea of adaptive and asymmetric computation can be a general solution for accelerating various architectures. We extend our method to hierarchical models including CNNs and hierarchical vision Transformers as well as more complex dense prediction tasks that require structured feature maps by formulating a more generic dynamic spatial sparsification framework with progressive sparsification and asymmetric computation for different spatial locations. By applying lightweight fast paths to less informative features and using more expressive slow paths to more important locations, we can maintain the structure of feature maps while significantly reducing the overall computations. Extensive experiments demonstrate the effectiveness of our framework on various modern architectures and different visual recognition tasks. Our results clearly demonstrate that dynamic spatial sparsification offers a new and more effective dimension for model acceleration. Code is available at <a class="link-external link-https" href="https://github.com/raoyongming/DynamicViT" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to accelerate Vision Transformers and Convolutional Neural Networks (CNNs) by leveraging spatial sparsity in visual data. Specifically:

1. **Dynamic Spatial Sparsification Framework**:
- A dynamic token sparsification framework is proposed to progressively remove redundant tokens from the input, thereby accelerating Vision Transformer models.
- This framework is based on the observation that the final prediction relies only on a subset of the most informative regions, which is sufficient for accurate image recognition.

2. **Adaptive and Asymmetric Computation**:
- It is found that adaptive and asymmetric computation can serve as a general solution for accelerating various architectures.
- The method is extended to more complex dense prediction tasks, including CNNs and hierarchical Vision Transformers.

3. **Experimental Validation**:
- Extensive experiments show that by hierarchically removing 66% of the input tokens, the method can significantly reduce 31%~35% of FLOPs and increase throughput by over 40%, with an accuracy drop of no more than 0.5%.
- For modern CNNs and Swin Transformers, introducing asymmetric computation can achieve similar acceleration effects.
- Satisfactory results are also achieved on complex tasks such as semantic segmentation and object detection.

In summary, the method proposed in the paper provides a new and more efficient dimension for model acceleration, demonstrating the potential of dynamic spatial sparsification in various visual tasks.