Abstract:We propose a novel algorithm for combined unit/filter and layer pruning of deep neural networks that functions during training and without requiring a pre-trained network to apply. Our algorithm optimally trades-off learning accuracy and pruning levels while balancing layer vs. unit/filter pruning and computational vs. parameter complexity using only three user-defined parameters, which are easy to interpret and tune. The optimal network structure is found as the solution of a stochastic optimization problem over the network weights and the parameters of variational Bernoulli distributions for 0/1 Random Variables scaling the units and layers of the network. Pruning occurs when a variational parameter converges to 0 rendering the corresponding structure permanently inactive, thus saving computations during training and prediction. A key contribution of our approach is to define a cost function that combines the objectives of prediction accuracy and network pruning in a computational/parameter complexity-aware manner and the automatic selection of the many regularization parameters. We show that the solutions of the optimization problem to which the algorithm converges are deterministic networks. We analyze the ODE system that underlies our stochastic optimization algorithm and establish domains of attraction around zero for the dynamics of the network parameters. These results provide theoretical support for safely pruning units/filters and/or layers during training and lead to practical pruning conditions. We evaluate our method on the CIFAR-10/100 and ImageNet datasets using ResNet architectures and demonstrate that our method improves upon layer only or unit only pruning and favorably competes with combined unit/filter and layer pruning algorithms requiring pre-trained networks with respect to pruning ratios and test accuracy.

Structure pruning strategies for min-max modular network

An Algorithm for Pruning Redundant Modules in Min-Ma Modular Network

An algorithm for pruning redundant modules in min-max modular network with GZC function

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

On Efficient Selection of Binary Classifiers for Min-Max Modular Classifier

Module Combination Based on Decision Tree in Min-Max Modular Network

Structural Pruning in Deep Neural Networks: A Small-World Approach

Pruning training samples using a supervised clustering algorithm

Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery

Multiple Sparse Spaces Network Pruning Via a Joint Similarity Criterion

Decomposition and Parallel Learning of Imbalanced Classification Problems by Min-Max Modular Neural Network

Improvement On Response Performance Of Min-Max Modular Classifier By Symmetric Module Selection

Network Pruning Spaces

On Effective Decomposition of Training Data Sets for Min-Max Modular Classifier

Emergent on-line learning in min-max modular neural networks

Filter Pruning for CNN with Enhanced Linear Representation Redundancy

A Min-Max Modular Network with Gaussian-Zero-Crossing Function

Task Decomposition and Module Combination Based on Class Relations: a Modular Neural Network for Pattern Classification

Graph Modularity: Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Neural Networks.