A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks
Jie Hu,Peng Lin,Huajun Zhang,Zining Lan,Wenxin Chen,Kailiang Xie,Siyun Chen,Hao Wang,Sheng Chang
DOI: https://doi.org/10.1109/access.2023.3267469
IF: 3.9
2023-04-25
IEEE Access
Abstract:It is well known that significant computational power and a large amount of memory are required for deep neural networks, which makes them difficult to apply in resource-limited environments. So, many network compression and acceleration technologies have emerged, of which connection pruning is widely applied due to its effectiveness and convenience. A novel connection pruning method for full model capacity on multiple sparse structures is proposed in this paper. We design a simple and efficient function called Dynamic Processing Unit (DPU) for handling the evaluated weights. Our method has the following features: 1) Instead of being pruned directly or set to 0, the weights are controlled by the DPU to determine whether they will be used during subsequent forward passes of the network during the iteration of pruning training. 2) It supports the traditional multi-steps prune method as well as the end-to-end training mode that can get a compressed network in a single stage by fusing training and pruning. 3) It can learn multiple useful sparse structures, including, but not limited to, depth-wise, filter-wise, channel-wise, 2D-filter-wise, row-wise, column-wise, connection-wise and mixed sparse structures. Our method is tested on various widely-used datasets and models, such as the LeNet and the ResNet on MNIST and CIFAR-10. Importantly, it demonstrates good performance in all these cases. Some details about our method can be found at the following URL: https://github.com/hujie369/DPU
computer science, information systems,telecommunications,engineering, electrical & electronic