A Simple and Effective Pruning Approach for Large Language Models

Mingjie Sun,Zhuang Liu,Anna Bair,J. Zico Kolter

2024-05-07

Abstract:As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at <a class="link-external link-https" href="https://github.com/locuslab/wanda" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

This paper attempts to address the challenges of pruning large language models (LLMs). Specifically:

1. **No Retraining Required**: Existing pruning methods typically require retraining the model or performing complex weight updates, which is impractical for large-scale LLMs due to the enormous computational resources needed. The proposed method, Wanda, does not require retraining or weight updates.
2. **Efficient Pruning**: The paper introduces a simple yet effective pruning method called Wanda, which evaluates the importance of weights by combining weight magnitude and input activation, and performs local comparisons on a per-output basis. This method induces sparsity in pre-trained LLMs with high computational efficiency, requiring only a single forward pass to complete.
3. **Addressing LLM Pruning Challenges**: Traditional pruning methods, such as weight magnitude pruning, perform poorly when dealing with LLMs, especially at low sparsity levels. Wanda proposes new pruning metrics to better adapt to the characteristics of LLMs.

Overall, this paper aims to develop an efficient, retraining-free LLM pruning method to overcome the limitations of existing methods on large-scale models.