BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

Wei Huang,Yangdong Liu,Haotong Qin,Ying Li,Shiming Zhang,Xianglong Liu,Michele Magno,Xiaojuan Qi

2024-05-15

Abstract:Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at

Machine Learning,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper aims to address the deployment challenges of large language models (LLMs) on memory-constrained devices, particularly the issues arising from the massive parameter sizes and computational demands of these models. To tackle these problems, the research team proposed BiLLM, a post-training quantization scheme specifically designed for pre-trained large language models. This scheme can compress model weights to just 1 bit, significantly reducing storage and computational resource requirements.

Specifically, BiLLM achieves this goal through the following key technical means:
1. **Structured Selection of Significant Weights**: Utilizing the Hessian matrix to identify weight elements that have a significant impact on model output and minimizing compression loss through an efficient binary residual approximation strategy.
2. **Optimal Partition Search for Insignificant Weights**: Considering the bell-shaped distribution characteristic of insignificant weights, an optimal partition search method is proposed to accurately group and binarize these weights, further reducing quantization error.
3. **Binary Residual Approximation**: For significant weights, a recursive computation strategy is employed to compensate for weight binarization, minimizing the binarization error of significant weights through a quadratic approximation approach.

Experimental results show that BiLLM not only achieves high-precision inference across various large language model families (e.g., achieving a perplexity of 8.41 on LLaMA2-70B) but also operates at an extremely low average bit-width (approximately 1.08 bits), significantly outperforming existing state-of-the-art quantization methods. Additionally, BiLLM demonstrates satisfactory timeliness, such as completing the binarization process of a 700 million parameter-scale LLM within half an hour on a single GPU.