Abstract:As a practical approach for compressing convolutional neural networks (CNNs), network pruning has been rapidly developed in recent years. The conventional methods prune inactive filters permanently from models to reduce the width of each layer and then train the pruned model until convergence. However, such methods have limitations in that: (1) The activation-based pruning criteria ignore the correlation between filters, leading to attenuation in types of features; (2) The permanent filter removal restricts the architecture of models in the subsequent training so that reducing the chances of learning more features; (3) The single-width compression may generate narrow layers that block the information flow, resulting in limited feature capacity in the next layers and hard optimization. These limitations reduce the feature diversity in the pruned model and thus lead to sub-optimal model quality. In this paper, a compression method named filter clustering is proposed to rectify the problem of poor feature diversity in traditional pruning and achieve better model quality from three perspectives. Firstly, to maintain the variety of features after pruning, we treat the model compression as a clustering task and merge filters with similar outputs, rather than removing inactive filters. Specifically, a handy estimation approach is designed to convert the similarity of the output into filter similarity, which liberates the measurement from sampling numerous images. Secondly, to increase the probability of learning more features during training, we propose a periodic training and clustering pipeline, which creates a larger optimization space by dynamically exploring different sub-model architectures. Finally, to prevent the feature capacity from being influenced by the narrow layers, we introduce and leverage a fusible anti-blocking branch to smoothly remove such layers. Extensive experiments demonstrate that the proposed method can achieve compact models with better feature diversity and reduce 1%~15% more calculations than the previous methods while maintaining performance.

Accelerating Deep Convnets Via Sparse Subspace Clustering

Deep Neural Network Acceleration with Sparse Prediction Layers

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Convolutional neural network acceleration algorithm based on filters pruning

Filter Pruning Via Feature Map Clustering.

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression.

A Novel Clustering-Based Filter Pruning Method for Efficient Deep Neural Networks.

Accelerating Convolutional Neural Networks By Group-Wise 2d-Filter Pruning

SCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners

A One-step Pruning-recovery Framework for Acceleration of Convolutional Neural Networks

Online Filter Weakening and Pruning for Efficient Convnets

Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks

Accelerating Convolutional Networks via Global & Dynamic Filter Pruning

Filter Clustering for Compressing CNN Model with Better Feature Diversity

Online Filter Clustering and Pruning for Efficient Convnets

Toward Compact ConvNets via Structure-Sparsity Regularized Filter Pruning

Weak sub-network pruning for strong and efficient neural networks

CSHE: Network Pruning by Using Cluster Similarity and Matrix Eigenvalues

Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks

Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks