Abstract:DNN pruning is a popular way to reduce the size of a model, improve the inference latency, and minimize the power consumption on DNN accelerators. However, existing approaches might be too complex, expensive or ineffective to apply to a variety of vision/language tasks, DNN architectures and to honor structured pruning constraints. In this paper, we propose an efficient yet effective train-time pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers state-of-the-art qualities in model size, accuracy, and training cost. PDP uses a dynamic function of weights during training to generate soft pruning masks for the weights in a parameter-free manner for a given pruning target. While differentiable, the simplicity and efficiency of PDP make it universal enough to deliver state-of-the-art random/structured/channel pruning results on various vision and natural language tasks. For example, for MobileNet-v1, PDP can achieve 68.2% top-1 ImageNet1k accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the state-of-the-art algorithms. Also, PDP yields over 83.1% accuracy on Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy. In addition, PDP can be applied to structured pruning, such as N:M pruning and channel pruning. For 1:4 structured pruning of ResNet18, PDP improved the top-1 ImageNet1k accuracy by over 3.6% over the state-of-the-art. For channel pruning of ResNet50, PDP reduced the top-1 ImageNet1k accuracy by 0.6% from the state-of-the-art.

Bayesian Based Re-parameterization for DNN Model Pruning

Class-Aware Pruning for Efficient Neural Networks

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Dynamical Channel Pruning by Conditional Accuracy Change for Deep Neural Networks

A Probabilistic Approach to Neural Network Pruning

Filter Pruning Via Feature Map Clustering.

Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization

Revisiting Random Channel Pruning for Neural Network Compression

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Exploiting Channel Similarity for Network Pruning.

Weight Reparametrization for Budget-Aware Network Pruning

Discrimination-Aware Channel Pruning For Deep Neural Networks

ACP: Automatic Channel Pruning Method by Introducing Additional Loss for Deep Neural Networks.

Variational Convolutional Neural Network Pruning

PDP: Parameter-free Differentiable Pruning is All You Need

Investigating Channel Pruning through Structural Redundancy Reduction -- A Statistical Study

AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance

Towards Optimal Filter Pruning with Balanced Performance and Pruning Speed.

Network Pruning Spaces

Adaptive Channel Pruning for Trainability Protection.