Abstract:Network pruning reduces the computation costs of an over-parameterized network without performance damage. Prevailing pruning algorithms pre-define the width and depth of the pruned networks, and then transfer parameters from the unpruned network to pruned networks. To break the structure limitation of the pruned networks, we propose to apply neural architecture search to search directly for a network with flexible channel and layer sizes. The number of the channels/layers is learned by minimizing the loss of the pruned networks. The feature map of the pruned network is an aggregation of K feature map fragments (generated by K networks of different sizes), which are sampled based on the probability distribution. The loss can be back-propagated not only to the network weights, but also to the parameterized distribution to explicitly tune the size of the channels/layers. Specifically, we apply channel-wise interpolation to keep the feature map with different channel sizes aligned in the aggregation procedure. The maximum probability for the size in each distribution serves as the width and depth of the pruned network, whose parameters are learned by knowledge transfer, e.g., knowledge distillation, from the original networks. Experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate the effectiveness of our new perspective of network pruning compared to traditional network pruning algorithms. Various searching and knowledge transfer approaches are conducted to show the effectiveness of the two components. Code is at: https://github.com/D-X-Y/NAS-Projects.

Edge-Wise One-Level Global Pruning on NAS Generated Networks

Class-Aware Pruning for Efficient Neural Networks

NAP: Neural architecture search with pruning

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Adaptive Search-and-Training for Robust and Efficient Network Pruning

Efficient Architecture Search via Bi-level Data Pruning

DDPNAS: Efficient Neural Architecture Search Via Dynamic Distribution Pruning

Neural Architecture Search via Trainless Pruning Algorithm: A Bayesian Evaluation of a Network with Multiple Indicators

Pruning Based Training-Free Neural Architecture Search

Pruning graph neural networks by evaluating edge properties

Data-Efficient Adaptive Global Pruning for Convolutional Neural Networks in Edge Computing

Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization

One-Shot Neural Architecture Search by Dynamically Pruning Supernet in Hierarchical Order

Cloud–Edge Collaborative Inference with Network Pruning

Network Pruning Via Transformable Architecture Search.

Network Automatic Pruning: Start NAP and Take a Nap

EMNAPE: Efficient Multi-Dimensional Neural Architecture Pruning for EdgeAI.

Tiered Pruning for Efficient Differentialble Inference-Aware Neural Architecture Search

AACP: Model Compression by Accurate and Automatic Channel Pruning.

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Beyond Network Pruning: a Joint Search-and-training Approach