Abstract:Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning and explore emerging topics, including pruning for large language models, large multimodal models, post-training pruning, and different supervision levels for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. To facilitate future research, we build a curated collection of datasets, networks, and evaluations on different applications. Finally, we provide valuable recommendations on selecting pruning methods and prospect several promising research directions. We build a repository at <a class="link-external link-https" href="https://github.com/hrcheng1066/awesome-pruning" rel="external noopener nofollow">this https URL</a>.

Pruning Training Corpus to Speedup Text Classification

An Approach to Improve Text Classification Efficiency.

Fast Text Classification: A Training-Corpus Pruning Based Approach

Online Text Classification

Toward Semi-Automatic Construction of Training-Corpus for Text Classification

Evaluation And Construction Of Training Corpuses For Text Classification: A Preliminary Study

Improving short text classification using public search engines

A Fast Post-Training Pruning Framework for Transformers

Class-Aware Pruning for Efficient Neural Networks

TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

Improving convolutional neural network for text classification by recursive data pruning

Gradient-based Intra-attention Pruning on Pre-trained Language Models

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

Text Quality-Based Pruning for Efficient Training of Language Models

Unit Feature Based Pruning of Large-Scale Speech Corpus Using Decision Tree

Block Pruning For Faster Transformers

Accelerating CNN Training by Pruning Activation Gradients

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Optimizing News Text Classification with Bi-LSTM and Attention Mechanism for Efficient Data Processing

Cleaner Pretraining Corpus Curation with Neural Web Scraping