Abstract:Filter pruning is a significant feature selection technique to shrink the existing feature fusion schemes (especially on convolution calculation and model size), which helps to develop more efficient feature fusion models while maintaining state-of-the-art performance. In addition, it reduces the storage and computation requirements of deep neural networks (DNNs) and accelerates the inference process dramatically. Existing methods mainly rely on manual constraints such as normalization to select the filters. A typical pipeline comprises two stages: first pruning the original neural network and then fine-tuning the pruned model. However, choosing a manual criterion can be somehow tricky and stochastic. Moreover, directly regularizing and modifying filters in the pipeline suffer from being sensitive to the choice of hyperparameters, thus making the pruning procedure less robust. To address these challenges, we propose to handle the filter pruning issue through one stage: using an attention-based architecture that adaptively fuses the filter selection with filter learning in a unified network. Specifically, we present a pruning method named adding before pruning (ABP) to make the model focus on the filters of higher significance by training instead of man-made criteria such as norm, rank, etc. First, we add an auxiliary attention layer into the original model and set the significance scores in this layer to be binary. Furthermore, to propagate the gradients in the auxiliary attention layer, we design a specific gradient estimator and prove its effectiveness for convergence in the graph flow through mathematical derivation. In the end, to relieve the dependence on the complicated prior knowledge for designing the thresholding criterion, we simultaneously prune and train the filters to automatically eliminate network redundancy with recoverability. Extensive experimental results on the two typical image classification benchmarks, CIFAR-10 and ILSVRC-2012, illustrate that the proposed approach performs favorably against previous state-of-the-art filter pruning algorithms.

Fpar: filter pruning via attention and rank enhancement for deep convolutional neural networks acceleration

Towards Efficient Filter Pruning Via Adaptive Automatic Structure Search

Adding Before Pruning: Sparse Filter Fusion for Deep Convolutional Neural Networks Via Auxiliary Attention

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

Class-Aware Pruning for Efficient Neural Networks

FPC: Filter Pruning Via the Contribution of Output Feature Map for Deep Convolutional Neural Networks Acceleration.

Filter Pruning Via Geometric Median for Deep Convolutional Neural Networks Acceleration

Filter Pruning via Probabilistic Model-based Optimization for Accelerating Deep Convolutional Neural Networks

Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks

A Filter Rank Based Pruning Method for Convolutional Neural Networks

Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration.

Filter Pruning Via Feature Map Clustering.

Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks

Data Agnostic Filter Gating for Efficient Deep Networks

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Non-Parametric Adaptive Network Pruning

NFP: A No Fine-tuning Pruning Approach for Convolutional Neural Network Compression

Network Pruning Using Adaptive Exemplar Filters

Learning Compact ConvNets Through Filter Pruning Based on the Saliency of a Feature Map.

FASS-pruner: Customizing a Fine-Grained CNN Accelerator-Aware Pruning Framework Via Intra-Filter Splitting and Inter-Filter Shuffling

Incremental Filter Pruning Via Random Walk for Accelerating Deep Convolutional Neural Networks