Abstract:Convolutional neural networks (CNNs) have developed to become powerful models for various computer vision tasks ranging from object detection to semantic segmentation. However, most of the state-of-the-art CNNs cannot be deployed directly on edge devices such as smartphones and drones, which need low latency under limited power and memory bandwidth. One popular, straightforward approach to compressing CNNs is network slimming, which imposes $\ell_1$ regularization on the channel-associated scaling factors via the batch normalization layers during training. Network slimming thereby identifies insignificant channels that can be pruned for inference. In this paper, we propose replacing the $\ell_1$ penalty with an alternative nonconvex, sparsity-inducing penalty in order to yield a more compressed and/or accurate CNN architecture. We investigate $\ell_p (0 < p < 1)$, transformed $\ell_1$ (T$\ell_1$), minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) due to their recent successes and popularity in solving sparse optimization problems, such as compressed sensing and variable selection. We demonstrate the effectiveness of network slimming with nonconvex penalties on three neural network architectures -- VGG-19, DenseNet-40, and ResNet-164 -- on standard image classification datasets. Based on the numerical experiments, T$\ell_1$ preserves model accuracy against channel pruning, $\ell_{1/2, 3/4}$ yield better compressed models with similar accuracies after retraining as $\ell_1$, and MCP and SCAD provide more accurate models after retraining with similar compression as $\ell_1$. Network slimming with T$\ell_1$ regularization also outperforms the latest Bayesian modification of network slimming in compressing a CNN architecture in terms of memory storage while preserving its model accuracy after channel pruning.

Overfitting Remedy by Sparsifying Regularization on Fully-Connected Layers of CNNs.

SparseConnect: Regularising CNNs on Fully Connected Layers

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

Efficient Network Compression Through Smooth-Lasso Constraint

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization

Towards Generalized Entropic Sparsification for Convolutional Neural Networks

Adaptive Structured Sparse Network for Efficient CNNs with Feature Regularization.

Prune the Convolutional Neural Networks with Sparse Shrink

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

Improving Network Slimming with Nonconvex Regularization

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

Learning Efficient Convolutional Networks Through Network Slimming.

Improving Deep Neural Network Sparsity Through Decorrelation Regularization

SlimConv: Reducing Channel Redundancy in Convolutional Neural Networks by Features Recombining

Multi-scale Convolution Aggregation and Stochastic Feature Reuse for DenseNets

Learning Sparse Patterns in Deep Neural Networks

Enhancing Generalization in Convolutional Neural Networks through Regularization with Edge and Line Features