Abstract:Vision transformers have emerged as a promising alternative to convolutional neural networks for various image analysis tasks, offering comparable or superior performance. However, one significant drawback of ViTs is their resource-intensive nature, leading to increased memory footprint, computation complexity, and power consumption. To democratize this high-performance technology and make it more environmentally friendly, it is essential to compress ViT models, reducing their resource requirements while maintaining high performance. In this paper, we introduce a new block-structured pruning to address the resource-intensive issue for ViTs, offering a balanced trade-off between accuracy and hardware acceleration. Unlike unstructured pruning or channel-wise structured pruning, block pruning leverages the block-wise structure of linear layers, resulting in more efficient matrix multiplications. To optimize this pruning scheme, our paper proposes a novel hardware-aware learning objective that simultaneously maximizes speedup and minimizes power consumption during inference, tailored to the block sparsity structure. This objective eliminates the need for empirical look-up tables and focuses solely on reducing parametrized layer connections. Moreover, our paper provides a lightweight algorithm to achieve post-training pruning for ViTs, utilizing second-order Taylor approximation and empirical optimization to solve the proposed hardware-aware objective. Extensive experiments on ImageNet are conducted across various ViT architectures, including DeiT-B and DeiT-S, demonstrating competitive performance with other pruning methods and achieving a remarkable balance between accuracy preservation and power savings. Especially, we achieve up to 3.93x and 1.79x speedups on dedicated hardware and GPUs respectively for DeiT-B, and also observe an inference power reduction by 1.4x on real-world GPUs.

A Comprehensive Study of Structural Pruning for Vision Models

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Isomorphic Pruning for Vision Models

Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression

GOHSP: A Unified Framework of Graph and Optimization-based Heterogeneous Structured Pruning for Vision Transformer

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

FreePrune: an Automatic Pruning Framework Across Various Granularities Based on Training-Free Evaluation

Automatic Hyper-Parameter Search for Vision Transformer Pruning

Structured Pruning for Deep Convolutional Neural Networks: A survey

SAViT: Structure-Aware Vision Transformer Pruning Via Collaborative Optimization.

Adaptive Activation-based Structured Pruning

LPViT: Low-Power Semi-structured Pruning for Vision Transformers

ThinResNet: A New Baseline for Structured Convolutional Networks Pruning

Pruning Foundation Models for High Accuracy without Retraining

Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions

ARPruning: An automatic channel pruning based on attention map ranking

Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint

Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience

Fluctuation-based Adaptive Structured Pruning for Large Language Models

RemoteTrimmer: Adaptive Structural Pruning for Remote Sensing Image Classification

UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer