Abstract:Post-training Sparsity (PTS) is a recently emerged avenue that chases efficient network sparsity with limited data in need. Existing PTS methods, however, undergo significant performance degradation compared with traditional methods that retrain the sparse networks via the whole dataset, especially at high sparsity ratios. In this paper, we attempt to reconcile this disparity by transposing three cardinal factors that profoundly alter the performance of conventional sparsity into the context of PTS. Our endeavors particularly comprise (1) A base-decayed sparsity objective that promotes efficient knowledge transferring from dense network to the sparse counterpart. (2) A reducing-regrowing search algorithm designed to ascertain the optimal sparsity distribution while circumventing overfitting to the small calibration set in PTS. (3) The employment of dynamic sparse training predicated on the preceding aspects, aimed at comprehensively optimizing the sparsity structure while ensuring training stability. Our proposed framework, termed UniPTS, is validated to be much superior to existing PTS methods across extensive benchmarks. As an illustration, it amplifies the performance of POT, a recently proposed recipe, from 3.9% to 68.6% when pruning ResNet-50 at 90% sparsity ratio on ImageNet. We release the code of our paper at <a class="link-external link-https" href="https://github.com/xjjxmu/UniPTS" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the case of limited resources, how to efficiently achieve post - training sparsity (PTS) of neural networks with a small amount of data and maintain the model performance. Specifically, the existing PTS methods perform poorly at high sparsity rates, especially when only a small amount of calibration data is available, and the model performance drops significantly. Therefore, this paper proposes a unified framework, UniPTS, aiming to improve this problem. ### Main Problems and Background 1. **Limitations of Existing PTS Methods**: - The performance of existing PTS methods drops significantly at high sparsity rates, especially when there is only a small amount of calibration data. - Traditional sparsification methods need to retrain the sparse network using the complete training set, which is unrealistic in the case of limited resources. 2. **Objectives**: - Propose a new framework, UniPTS, which can efficiently achieve the sparsification of neural networks with a small amount of data in the case of limited resources and maintain or improve the model performance. ### Solutions To achieve this goal, UniPTS has been improved in three aspects: 1. **Base - Decayed Sparsity Objective**: - Traditional methods use the mean - squared error (MSE) as a supervision signal, but it is easy to introduce bias at high sparsity rates. - UniPTS introduces the global Kullback - Leibler (KL) divergence and optimizes the sparsity objective by dynamically adjusting the logarithm base, so as to better transfer the knowledge of the dense network to the sparse network. 2. **Reducing - Regrowing Sparsity Distribution Search**: - Use an evolutionary algorithm to search for the optimal inter - layer sparsity rate allocation to avoid over - fitting a small - scale calibration data set. - Introduce an excessive sparsity mechanism and a fitness evaluation of noise perturbation to ensure the robustness and effectiveness of the search process. 3. **Dynamic Sparse Training**: - Dynamically adjust the pruning and regrowing of weights during the training process to avoid the limitations brought by the fixed topology. - Use the straight - through estimator to approximate the gradients of pruned and unpruned weights to ensure training stability. ### Experimental Results The experimental results show that UniPTS significantly outperforms the existing PTS methods in multiple benchmark tests, especially at high sparsity rates. For example, on the ImageNet data set, UniPTS improves the accuracy of ResNet - 50 at a 90% sparsity rate from 3.9% to 68.6%. ### Summary By improving the sparsity objective, sparsity distribution search, and sparse training strategies, UniPTS successfully solves the problem of performance degradation of existing PTS methods at high sparsity rates and provides an effective solution for model compression in resource - limited environments.

UniPTS: A Unified Framework for Proficient Post-Training Sparsity

Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes

SUBP: Soft Uniform Block Pruning for 1xn Sparse CNNs Multithreading Acceleration

SUBP: Soft Uniform Block Pruning for 1 X N Sparse CNNs Multithreading Acceleration

PTSBench: A Comprehensive Post-Training Sparsity Benchmark Towards Algorithms and Models

MaxQ: Multi-Axis Query for N:M Sparsity Network

Enhanced Sparsification via Stimulative Training

Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning

OptG: Optimizing Gradient-driven Criteria in Network Sparsity

PRUNING IN TRAINING: LEARNING AND RANKING SPARSE CONNECTIONS IN DEEP CONVOLUTIONAL NETWORKS

Dynamic Sparsity Is Channel-Level Sparsity Learner

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

PrUE: Distilling Knowledge from Sparse Teacher Networks

MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge

Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems

Dual sparse training framework: inducing activation map sparsity via Transformed $\ell1$ regularization

Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

Learning sparse tag patterns for social image classification

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks