Abstract:Structured pruning is a commonly used convolutional neural network (CNN) compression approach. Pruning rate setting is a fundamental problem in structured pruning. Most existing works introduce too many additional learnable parameters to assign different pruning rates across different layers in CNN or cannot control the compression rate explicitly. Since too narrow network blocks information flow for training, automatic pruning rate setting cannot explore a high pruning rate for a specific layer. To overcome these limitations, we propose a novel framework named Layer Adaptive Progressive Pruning (LAPP), which gradually compresses the network during initial training of a few epochs from scratch. In particular, LAPP designs an effective and efficient pruning strategy that introduces a learnable threshold for each layer and FLOPs constraints for network. Guided by both task loss and FLOPs constraints, the learnable thresholds are dynamically and gradually updated to accommodate changes of importance scores during training. Therefore the pruning strategy can gradually prune the network and automatically determine the appropriate pruning rates for each layer. What's more, in order to maintain the expressive power of the pruned layer, before training starts, we introduce an additional lightweight bypass for each convolutional layer to be pruned, which only adds relatively few additional burdens. Our method demonstrates superior performance gains over previous compression methods on various datasets and backbone architectures. For example, on CIFAR-10, our method compresses ResNet-20 to 40.3% without accuracy drop. 55.6% of FLOPs of ResNet-18 are reduced with 0.21% top-1 accuracy increase and 0.40% top-5 accuracy increase on ImageNet.

Iterative clustering pruning for convolutional neural networks

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Automatic channel pruning via clustering and swarm intelligence optimization for CNN

Global balanced iterative pruning for efficient convolutional neural networks

Regularized Training Framework for Combining Pruning and Quantization to Compress Neural Networks

Pruning-and-distillation: One-stage Joint Compression Framework for CNNs Via Clustering

CCPrune: Collaborative Channel Pruning for Learning Compact Convolutional Networks

Energy-saving CNN with Clustering Channel Pruning

Where to Prune: Using LSTM to Guide Data-Dependent Soft Pruning

BAP: Bilateral Asymptotic Pruning for Optimizing CNNs on Image Tasks

Cross-layer importance evaluation for neural network pruning

Frequency-Domain Dynamic Pruning for Convolutional Neural Networks

Loss-Driven Channel Pruning of Convolutional Neural Networks.

Compressing CNNs Using Multilevel Filter Pruning for the Edge Nodes of Multimedia Internet of Things

Cluster, Reconstruct and Prune: Equivalent Filter Pruning for CNNs Without Fine-Tuning.

LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch

Channel Pruning Based on Mean Gradient for Accelerating Convolutional Neural Networks

A Clustering Pruning Method Based on Multidimensional Channel Information

Pruning filters with L1-norm and standard deviation for CNN compression