Abstract:Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. However, assessing the relative importance of different sub-structures remains a significant challenge, particularly in advanced vision models featuring novel mechanisms and architectures like self-attention, depth-wise convolutions, or residual connections. These heterogeneous substructures usually exhibit diverged parameter scales, weight distributions, and computational topology, introducing considerable difficulty to importance comparison. To overcome this, we present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures such as Vision Transformers and CNNs, and delivers competitive performance across different model sizes. Isomorphic Pruning originates from an observation that, when evaluated under a pre-defined importance criterion, heterogeneous sub-structures demonstrate significant divergence in their importance distribution, as opposed to isomorphic structures that present similar importance patterns. This inspires us to perform isolated ranking and comparison on different types of sub-structures for more reliable pruning. Our empirical results on ImageNet-1K demonstrate that Isomorphic Pruning surpasses several pruning baselines dedicatedly designed for Transformers or CNNs. For instance, we improve the accuracy of DeiT-Tiny from 74.52% to 77.50% by pruning an off-the-shelf DeiT-Base model. And for ConvNext-Tiny, we enhanced performance from 82.06% to 82.18%, while reducing the number of parameters and memory usage. Code is available at \url{<a class="link-external link-https" href="https://github.com/VainF/Isomorphic-Pruning" rel="external noopener nofollow">this https URL</a>}.

Hierarchical Model Compression Via Shape-Edge Representation of Feature Maps—an Enlightenment from the Primate Visual System

Class-Aware Pruning for Efficient Neural Networks

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Towards Efficient Filter Pruning Via Topology

Towards Efficient Filter Pruning Via Adaptive Automatic Structure Search

MCMC: Multi-Constrained Model Compression Via One-Stage Envelope Reinforcement Learning.

Filter Pruning Via Feature Map Clustering.

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Editorial for Pattern Recognition Letters Special Issue on Face-Based Emotion Understanding

Pruning at a Glance: Global Neural Pruning for Model Compression

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Where to Prune: Using LSTM to Guide Data-Dependent Soft Pruning

Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience

Resource Efficient Neural Networks Using Hessian Based Pruning

Model pruning based on filter similarity for edge device deployment

UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer

Cloud–Edge Collaborative Inference with Network Pruning

Isomorphic Pruning for Vision Models

Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression

Where to Prune: Using LSTM to Guide End-to-end Pruning