Abstract:Pruning methods to compress and accelerate deep convolutional neural networks (CNNs) have recently attracted growing attention, with the view of deploying pruned networks on resource-constrained hardware devices. However, most existing methods focus on small granularities, such as weight, kernel and filter, for the exploration of pruning. Thus, it will be bound to iteratively prune the whole neural networks based on those small granularities for high compression ratio with little performance loss. To address these issues, we theoretically analyze the relationship between the activation and gradient sparsity, and the channel saliency. Based on our findings, we propose a novel and effective method of weak sub-network pruning (WSP). Specifically, for a well-trained network model, we divide the whole compression process into two non-iterative stages. The first stage is to directly obtain a strong sub-network by pruning the weakest one. We first identify the less important channels from all the layers and determine the weakest sub-network, whereby each selected channel makes a minimal contribution to both the feed-forward and feed-backward processes. Then, a one-shot pruning strategy is executed to form a strong sub-network enabling fine tuning, while significantly reducing the impact of the network depth and width on the compression efficiency, especially for deep and wide network architectures. The second stage is to globally fine-tune the strong sub-network using several epochs to restore its original recognition accuracy. Furthermore, our proposed method impacts on the fully-connected layers as well as the convolutional layers for simultaneous compression and acceleration. Comprehensive experiments on VGG16 and ResNet-50 involving a variety of popular benchmarks, such as ImageNet-1K, CIFAR-10, CUB-200 and PASCAL VOC, demonstrate that our WSP method achieves superior performance on classification, domain adaption and object detection tasks with small model size. Our source code is available at https://github.com/QingbeiGuo/WSP.git.

SUBP: Soft Uniform Block Pruning for 1xn Sparse CNNs Multithreading Acceleration

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

MaxQ: Multi-Axis Query for N:M Sparsity Network

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration.

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

1×N Block Pattern for Network Sparsity

Joint Sparsity with Mixed Granularity for Efficient GPU Implementation

1$\Times$n Block Pattern for Network Sparsity

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization.

Adaptive Pixel-wise Structured Sparse Network for Efficient CNNs

An Efficient CNN Training Accelerator Leveraging Transposable Block Sparsity

A CNN Channel Pruning Low-Bit Framework Using Weight Quantization with Sparse Group Lasso Regularization.

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization

One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget

1xN Pattern for Pruning Convolutional Neural Networks

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

Attention-based Adaptive Structured Continuous Sparse Network Pruning

Efficient Neural Networks with Spatial Wise Sparsity Using Unified Importance Map.

Weak sub-network pruning for strong and efficient neural networks

StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs

Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation