Abstract:Structured pruning is a commonly used convolutional neural network (CNN) compression approach. Pruning rate setting is a fundamental problem in structured pruning. Most existing works introduce too many additional learnable parameters to assign different pruning rates across different layers in CNN or cannot control the compression rate explicitly. Since too narrow network blocks information flow for training, automatic pruning rate setting cannot explore a high pruning rate for a specific layer. To overcome these limitations, we propose a novel framework named Layer Adaptive Progressive Pruning (LAPP), which gradually compresses the network during initial training of a few epochs from scratch. In particular, LAPP designs an effective and efficient pruning strategy that introduces a learnable threshold for each layer and FLOPs constraints for network. Guided by both task loss and FLOPs constraints, the learnable thresholds are dynamically and gradually updated to accommodate changes of importance scores during training. Therefore the pruning strategy can gradually prune the network and automatically determine the appropriate pruning rates for each layer. What's more, in order to maintain the expressive power of the pruned layer, before training starts, we introduce an additional lightweight bypass for each convolutional layer to be pruned, which only adds relatively few additional burdens. Our method demonstrates superior performance gains over previous compression methods on various datasets and backbone architectures. For example, on CIFAR-10, our method compresses ResNet-20 to 40.3% without accuracy drop. 55.6% of FLOPs of ResNet-18 are reduced with 0.21% top-1 accuracy increase and 0.40% top-5 accuracy increase on ImageNet.

Compact Neural Networks via Stacking Designed Basic Units

Class-Aware Pruning for Efficient Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Efficient Structure Slimming for Spiking Neural Networks

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression

Adversarial Structured Neural Network Pruning

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Structured Term Pruning for Computational Efficient Neural Networks Inference

An efficient pruning and fine-tuning method for deep spiking neural network

ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning

Structural Pruning in Deep Neural Networks: A Small-World Approach

Network Automatic Pruning: Start NAP and Take a Nap

Knapsack Pruning with Inner Distillation

Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch

AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

Pruning at a Glance: Global Neural Pruning for Model Compression