Abstract:Structured pruning is one of the most popular approaches to effectively compress the heavy deep neural networks (DNNs) into compact sub-networks while retaining performance. The existing methods suffer from multi-stage procedures along with significant engineering efforts and human expertise. The Only-Train-Once (OTO) series has been recently proposed to resolve the many pain points by streamlining the workflow by automatically conducting (i) search space generation, (ii) structured sparse optimization, and (iii) sub-network construction. However, the built-in sparse optimizers in the OTO series, i.e., the Half-Space Projected Gradient (HSPG) family, have limitations that require hyper-parameter tuning and the implicit controls of the sparsity exploration, consequently requires intervening by human expertise. To address such limitations, we propose a Hybrid Efficient Structured Sparse Optimizer (HESSO). HESSO could automatically and efficiently train a DNN to produce a high-performing subnetwork. Meanwhile, it is almost tuning-free and enjoys user-friendly integration for generic training applications. To address another common issue of irreversible performance collapse observed in pruning DNNs, we further propose a Corrective Redundant Identification Cycle (CRIC) for reliably identifying indispensable structures. We numerically demonstrate the efficacy of HESSO and its enhanced version HESSO-CRIC on a variety of applications ranging from computer vision to natural language processing, including large language model. The numerical results showcase that HESSO can achieve competitive even superior performance to varying state-of-the-arts and support most DNN architectures. Meanwhile, CRIC can effectively prevent the irreversible performance collapse and further enhance the performance of HESSO on certain applications. The code is available at <a class="link-external link-https" href="https://github.com/microsoft/only_train_once" rel="external noopener nofollow">this https URL</a>.

Learning Optimized Structure of Neural Networks by Hidden Node Pruning with L1 Regularization

Class-Aware Pruning for Efficient Neural Networks

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Structured Pruning for Efficient Convolutional Neural Networks Via Incremental Regularization

Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Efficient Structure Slimming for Spiking Neural Networks

LSOP: Layer-Scaled One-shot Pruning

Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error

Structured Pruning of Recurrent Neural Networks through Neuron Selection

Sparse optimization guided pruning for neural networks

Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method

The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Students and teachers learning together: a robust training strategy for neural network pruning

HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Network Pruning Spaces

Manifold Regularized Dynamic Network Pruning.

Learned Threshold Pruning