Abstract:Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. However, assessing the relative importance of different sub-structures remains a significant challenge, particularly in advanced vision models featuring novel mechanisms and architectures like self-attention, depth-wise convolutions, or residual connections. These heterogeneous substructures usually exhibit diverged parameter scales, weight distributions, and computational topology, introducing considerable difficulty to importance comparison. To overcome this, we present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures such as Vision Transformers and CNNs, and delivers competitive performance across different model sizes. Isomorphic Pruning originates from an observation that, when evaluated under a pre-defined importance criterion, heterogeneous sub-structures demonstrate significant divergence in their importance distribution, as opposed to isomorphic structures that present similar importance patterns. This inspires us to perform isolated ranking and comparison on different types of sub-structures for more reliable pruning. Our empirical results on ImageNet-1K demonstrate that Isomorphic Pruning surpasses several pruning baselines dedicatedly designed for Transformers or CNNs. For instance, we improve the accuracy of DeiT-Tiny from 74.52% to 77.50% by pruning an off-the-shelf DeiT-Base model. And for ConvNext-Tiny, we enhanced performance from 82.06% to 82.18%, while reducing the number of parameters and memory usage. Code is available at \url{<a class="link-external link-https" href="https://github.com/VainF/Isomorphic-Pruning" rel="external noopener nofollow">this https URL</a>}.

Large-scale Dataset Pruning with Dynamic Uncertainty

Class-Aware Pruning for Efficient Neural Networks

Dataset Pruning: Reducing Training Data by Examining Generalization Influence

Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolation

Confident magnitude-based neural network pruning

A Study in Dataset Pruning for Image Super-Resolution

Not All Data Matters: An End-to-End Adaptive Dataset Pruning Framework for Enhancing Model Performance and Efficiency

One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget

Isomorphic Pruning for Vision Models

Manifold Regularized Dynamic Network Pruning

Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuning

Efficient Training Acceleration via Sample-Wise Dynamic Probabilistic Pruning

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance

CRISP: Hybrid Structured Sparsity for Class-aware Model Pruning

Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks

Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning

How Does Pruning Impact Long-Tailed Multi-Label Medical Image Classifiers?

Prune it Yourself: Automated Pruning by Multiple Level Sensitivity

Students and teachers learning together: a robust training strategy for neural network pruning