Abstract:Deep learning models have evolved into powerful tools that can be used for many artificial intelligence tasks. However, deploying deep neural networks into real-world applications is still challenging due to their high computational complexity and storage overhead. Fortunately, a densely connected neural network can be converted into a sparsely connected network with low resource demand by the neural network compression. Since deep neural networks are complicated, compression mechanism should find a tradeoff between compression ratio and model accuracy. In this article, by analyzing the statistics of channel connection, we propose an interactive neural network compression mechanism including out-in-channel pruning and neural network quantization. Many channel pruning works apply structured sparsity regularization on each layer separately. We consider correlations between successive layers to retain predictive power of the compact network. A global greedy pruning algorithm is designed to remove redundant out-in-channels in an iterative way. Moreover, in order to solve the shortcomings of the one-shot quantization, we propose the incremental quantization algorithm in the dimension of the output channel, which can smooth network fluctuations and recover accuracy better during retraining. Our mechanism is comprehensively evaluated with various Convolutional Neural Networks (CNN) architectures on popular datasets. Notably, on ImageNet-1K, the out-in-channel pruning reduce 54.0 percent FLOPS on AlexNet and 50.0 percent FLOPs on ResNet-50 with only 0.15 and 0.37 percent top-1 accuracy drop respectively. On classification and style transfer tasks, the superiority of incremental quantization increases with the decrease of the number of quantization bits.

Structured Pruning Via Feature Channels Similarity and Mutual Learning for Convolutional Neural Network Compression

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Learning Compact Networks Via Similarity-Aware Channel Pruning

Structured Feature Sparsity Training for Convolutional Neural Network Compression

Towards Compact CNNs via Collaborative Compression

Efficient Network Compression Through Smooth-Lasso Constraint

Compressing Convolutional Neural Networks Via Factorized Convolutional Filters

Automatic channel pruning via clustering and swarm intelligence optimization for CNN

Exploiting Channel Similarity for Network Pruning.

Learning Low Resource Consumption CNN through Pruning and Quantization

Iterative clustering pruning for convolutional neural networks

NFP: A No Fine-tuning Pruning Approach for Convolutional Neural Network Compression

Filter Pruning with a Feature Map Entropy Importance Criterion for Convolution Neural Networks Compressing

Filter Pruning for CNN with Enhanced Linear Representation Redundancy

Efficient Similarity-based Passive Filter Pruning for Compressing CNNs

CSHE: Network Pruning by Using Cluster Similarity and Matrix Eigenvalues

Compression of Convolutional Neural Networks With Divergent Representation of Filters

LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch