Abstract:Abstract Deep neural networks (DNNs) have been applied in many pattern recognition or object detection applications. DNNs generally consist of millions or even billions of parameters. These demanding computational storage and requirements impede deployments of DNNs in resource-limited devices, such as mobile devices, micro-controllers. Simplification techniques such as pruning have commonly been used to slim DNN sizes. Pruning approaches generally quantify the importance of each component such as network weight. Weight values or weight gradients in training are commonly used as the importance metric. Small weights are pruned and large weights are kept. However, small weights are possible to be connected with significant weights which have impact to DNN outputs. DNN accuracy can be degraded significantly after the pruning process. This paper proposes a roulette wheel-like pruning algorithm, in order to simplify a trained DNN while keeping the DNN accuracy. The proposed algorithm generates a branch of pruned DNNs which are generated by a roulette wheel operator. Similar to the roulette wheel selection in genetic algorithms, small weights are more likely to be pruned but they can be kept; large weights are more likely to be kept but they can be pruned. The slimmest DNN with the best accuracy is selected from the branch. The performance of the proposed pruning algorithm is evaluated by two deterministic datasets and four non-deterministic datasets. Experimental results show that the proposed pruning algorithm generates simpler DNNs while DNN accuracy can be kept, compared to several existing pruning approaches.

Pruning Deep Neural Networks by Optimal Brain Damage

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Class-Aware Pruning for Efficient Neural Networks

Magnificent Minified Models

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Efficient Structure Slimming for Spiking Neural Networks

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Neural network relief: a pruning algorithm based on neural activity

Rethinking Pruning for Backdoor Mitigation: An Optimization Perspective

(Pen-) Ultimate DNN Pruning

Brain-Inspired Efficient Pruning: Exploiting Criticality in Spiking Neural Networks

A roulette wheel-based pruning method to simplify cumbersome deep neural networks

Structural Pruning in Deep Neural Networks: A Small-World Approach

Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks

Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

Adversarial Neuron Pruning Purifies Backdoored Deep Models.

Statistical Mechanical Analysis of Neural Network Pruning

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks