Abstract:Deep Neural Networks are an important class of machine learning algorithms that have demonstrated state-of-the-art accuracy for different cognitive tasks like image and speech recognition. Modem deep networks have millions to billions of parameters, which leads to high memory and energy requirements during training as well as during inference on resource-constrained edge devices. Consequently, pruning techniques have been proposed that remove less significant weights in deep networks, thereby reducing their memory and computational requirements. Pruning is usually performed after training the original network, and is followed by further retraining to compensate for the accuracy loss incurred during pruning. The prune-and-retrain procedure is repeated iteratively until an optimum tradeoff between accuracy and efficiency is reached. However, such iterative retraining adds to the overall training complexity of the network. In this work, we propose a dynamic pruning-while-training procedure, wherein we prune filters of the convolutional layers of a deep neural network during training itself, thereby precluding the need for separate retraining. We evaluate our dynamic pruning-while-training approach with three different pre-existing pruning strategies, viz. mean activation-based pruning, random pruning, and L1 normalization-based pruning. Our results for VGG-16 trained on CIFAR10 shows that L1 normalization provides the best performance among all the techniques explored in this work with less than 1% drop in accuracy after pruning 80% of the filters compared to the original network. We further evaluated the L1 normalization based pruning mechanism on CIFAR100. Results indicate that pruning while training yields a compressed network with almost no accuracy loss after pruning 50% of the filters compared to the original network and ~5% loss for high pruning rates (> 80%). The proposed pruning methodology yields 41% reduction in the number of computations and memory accesses during training for CIFAR10, CIFAR100 and ImageNet compared to training with retraining for 10 epochs.

Incremental Filter Pruning Via Random Walk for Accelerating Deep Convolutional Neural Networks

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Towards Efficient Filter Pruning Via Adaptive Automatic Structure Search

Towards Efficient Filter Pruning Via Topology

Filter Pruning via Probabilistic Model-based Optimization for Accelerating Deep Convolutional Neural Networks

Convolutional neural network acceleration algorithm based on filters pruning

Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks

Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks

Filter Pruning Via Feature Map Clustering.

Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration.

Pruning Filters while Training for Efficiently Optimizing Deep Learning Networks

Evolving Filter Criteria for Randomly Initialized Network Pruning in Image Classification

Accelerating Convolutional Neural Networks By Group-Wise 2d-Filter Pruning

Filter Pruning Via Geometric Median for Deep Convolutional Neural Networks Acceleration

Enhancing CNN Efficiency Through Mutual Information-Based Filter Pruning

Learning Compact ConvNets Through Filter Pruning Based on the Saliency of a Feature Map.

FPC: Filter Pruning Via the Contribution of Output Feature Map for Deep Convolutional Neural Networks Acceleration.

Accelerating Convolutional Networks via Global & Dynamic Filter Pruning

A Graphical Approach for Filter Pruning by Exploring the Similarity Relation between Feature Maps