Abstract:$N{:}M$ sparsity is an emerging model compression method supported by more and more accelerators to speed up sparse matrix multiplication in deep neural networks. Most existing $N{:}M$ sparsity methods compress neural networks with a uniform setting for all layers in a network or heuristically determine the layer-wise configuration by considering the number of parameters in each layer. However, very few methods have been designed for obtaining a layer-wise customized $N{:}M$ sparse configuration for vision transformers (ViTs), which usually consist of transformer blocks involving the same number of parameters. In this work, to address the challenge of selecting suitable sparse configuration for ViTs on $N{:}M$ sparsity-supporting accelerators, we propose ELSA, Exploiting Layer-wise $N{:}M$ Sparsity for ViTs. Considering not only all $N{:}M$ sparsity levels supported by a given accelerator but also the expected throughput improvement, our methodology can reap the benefits of accelerators supporting mixed sparsity by trading off negligible accuracy loss with both memory usage and inference time reduction for ViT models. For instance, our approach achieves a noteworthy 2.9$\times$ reduction in FLOPs for both Swin-B and DeiT-B with only a marginal degradation of accuracy on ImageNet. Our code will be released upon paper acceptance.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of how to select an appropriate layer - wise N:M sparse configuration for Vision Transformers (ViTs) to achieve acceleration. Specifically, the paper proposes solutions to the following challenges: 1. **Limitations of existing methods**: - Most existing N:M sparsification methods use a uniform setting for all network layers or heuristic configurations based on the number of parameters per layer. These methods have limited effectiveness when dealing with ViTs because each Transformer block in ViTs usually has the same number of parameters. 2. **Support for hardware accelerators**: - Modern hardware accelerators are starting to support mixed N:M sparsity (e.g., 2:4, 4:8, etc.), which can significantly accelerate sparse matrix multiplication. However, how to fully utilize these hardware features to optimize the performance of ViTs remains a challenge. 3. **Trade - off between accuracy and efficiency**: - Reducing memory usage and inference time while maintaining model accuracy is a key objective. Over - aggressive sparsification may lead to a significant drop in accuracy, so a balanced method needs to be found. To this end, the paper proposes ELSA (Exploiting Layer - wise N:M Sparsity for Vision Transformer Acceleration), a framework for exploring layer - wise N:M sparse configurations in ViTs. By considering all N:M sparse levels supported by the accelerator and the expected throughput improvement, ELSA can significantly reduce FLOPs (floating - point operations) with almost no impact on accuracy, thereby increasing the inference speed and reducing memory footprint. ### Specific contributions - **First exploration of layer - wise N:M sparse configurations in ViTs**: Research has been carried out on linear modules (such as linear projections and multi - layer perceptron layers). - **High - precision search for sparse configurations**: Not only the application accuracy but also the hardware efficiency is considered, and multiple efficient sparse configurations with different compression ratios can be obtained. - **Significant reduction in FLOPs**: For example, on the ImageNet classification task, ELSA achieves a 2.9 - fold reduction in FLOPs for the DeiT - B model while only causing a slight drop in accuracy. Through these contributions, ELSA provides new ideas and technical means for the efficient deployment of vision transformers.

ELSA: Exploiting Layer-wise N:M Sparsity for Vision Transformer Acceleration

Deep Neural Network Acceleration with Sparse Prediction Layers

Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer

Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

DynaSlim: Dynamic Slimming for Vision Transformers.

Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training.

An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers

SPViT: Enabling Faster Vision Transformers Via Latency-Aware Soft Token Pruning

Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference

CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Vision Transformer with Sparse Scan Prior

Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers

Pruning Self-attentions into Convolutional Layers in Single Path

Network Expansion for Practical Training Acceleration

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs