ARB-LLM: Alternating Refined Binarizations for Large Language Models

Zhiteng Li,Xianglong Yan,Tianao Zhang,Haotong Qin,Dong Xie,Jiang Tian,zhongchao shi,Linghe Kong,Yulun Zhang,Xiaokang Yang
2024-10-10
Abstract:Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_\text{X}$ and ARB-LLM$_\text{RC}$ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$_\text{RC}$ is the first to surpass FP16 models of the same size. The code and models will be available at <a class="link-external link-https" href="https://github.com/ZHITENGLI/ARB-LLM" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Machine Learning
What problem does this paper attempt to address?
The problems that this paper attempts to solve are the two major challenges encountered by large - language models (LLMs) during compression and quantization: 1. **Distribution Gap Problem**: Existing binarization methods have difficulty in reducing the distribution gap between binarized weights and full - precision weights. This gap leads to significant quantization errors, thereby affecting model performance. 2. **Column Deviation Problem**: There are obvious column deviations in the weight matrices of large - language models, and traditional row - vectorization methods cannot effectively handle these deviations. To solve these problems, the authors propose ARB - LLM, a new 1 - bit post - training quantization (PTQ) technique specifically designed for large - language models. Specifically, the main contributions of the paper include: - **Alternating Refined Binarization (ARB) Algorithm**: By gradually updating binarization parameters to align the distribution between binarized weights and full - precision weights, quantization errors are significantly reduced. - **Extended Versions ARB - X and ARB - RC**: - ARB - X introduces calibration data to enhance the effect of quantization parameter updates. - ARB - RC alternately refines the scaling factors along the row and column axes to deal with column deviations in the weight matrix. - **Column - Group Bitmap (CGB) Strategy**: By combining significant column bitmaps and group bitmaps, bitmap utilization is optimized, further improving performance. Through these improvements, ARB - LLM RC surpasses FP16 models of the same scale for the first time and significantly outperforms existing binarization methods in zero - sample question - answering tasks. ### Formula Summary - **Initial Quantization Error**: \[ L_1=\|W - cW\|_F^2, \quad \text{where} \quad cW = \alpha B+\mu \] - **Refined Mean Update**: \[ \mu_{\text{refine}}=\mu+\delta\mu, \quad \text{where} \quad \delta\mu=\frac{1}{m}\sum_{j = 1}^{m}R_j \] - **Refined Scaling Factor and Binary Matrix Update**: \[ \alpha_{\text{refine}}=\frac{1}{m}\text{diag}(B^{\top}(W-\mu_{\text{refine}})), \quad B_{\text{refine}}=\text{sign}(W-\mu_{\text{refine}}) \] - **Second - Order Binary Matrix**: \[ cW=\alpha_1B_1+\alpha_2B_2+\mu \] These formulas show how to minimize quantization errors by iteratively updating parameters, thereby improving the performance of binarized models.