Abstract:Modern Convolutional Neural Networks (CNNs) are usually restricted by their massive computation and high storage. Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degradation. Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factors, which neglects the fragility of the expressiveness of CNNs, and thus calls for a more gentle regularization scheme so that the networks can adapt during pruning. To achieve this, we propose a novel regularization-based pruning method, named IncReg, to incrementally assign different regularization factors to different weights based on their relative importance. Empirical analysis on CIFAR-10 dataset verifies the merits of IncReg. Further extensive experiments with popular CNNs on CIFAR-10 and ImageNet datasets show that IncReg achieves comparable to even better results compared with state-of-the-arts. Moreover, to resolve the problem that column pruning cannot be directly applied to off-the-shelf deep learning libraries for acceleration, we generalize IncReg from column pruning to spatial pruning, which can equip existing structured pruning methods (such as channel pruning) for further acceleration with ignorable accuracy loss. Our source codes and trained models are available at: https://github.com/mingsun-tse/caffe_increghttps://github.com/mingsun-tse/caffe_increg.

Deep Neural Architecture Pruning

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Class-Aware Pruning for Efficient Neural Networks

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Adversarial Structured Neural Network Pruning

Filter Pruning Via Feature Map Clustering.

Identifying and Pruning Redundant Structures for Deep Neural Networks

Learning to Prune Filters in Convolutional Neural Networks

Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy

Modeling of Pruning Techniques for Deep Neural Networks Simplification

Pruning filters with L1-norm and standard deviation for CNN compression

Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks

Frequency-Domain Dynamic Pruning for Convolutional Neural Networks

Cross-layer importance evaluation for neural network pruning

Pruning the Deep Neural Network by Similar Function

Towards Thinner Convolutional Neural Networks Through Gradually Global Pruning

Prune it Yourself: Automated Pruning by Multiple Level Sensitivity