Abstract:Large language models (LLMs) achieve remarkable performance in natural language understanding but require substantial computation and memory resources. Post-training quantization (PTQ) is a powerful compression technique extensively investigated in LLMs. However, existing PTQ methods are still not ideal in terms of accuracy and efficiency, especially with below 4 bit-widths. Standard PTQ methods using group-wise quantization suffer difficulties in quantizing LLMs accurately to such low-bit, but advanced methods remaining high-precision weights element-wisely are hard to realize their theoretical hardware efficiency. This paper presents a Salience-Driven Mixed-Precision Quantization scheme for LLMs, namely SliM-LLM. The scheme exploits the salience distribution of weights to determine optimal bit-width and quantizers for accurate LLM quantization, while aligning bit-width partition to groups for compact memory usage and fast integer inference. Specifically, the proposed SliM-LLM mainly relies on two novel techniques: (1) Salience-Determined Bit Allocation utilizes the clustering characteristics of salience distribution to allocate the bit-widths of each group, increasing the accuracy of quantized LLMs and maintaining the inference efficiency; (2) Salience-Weighted Quantizer Calibration optimizes the parameters of the quantizer by considering the element-wise salience within the group, balancing the maintenance of salient information and minimization of errors. Comprehensive experiments show that SliM-LLM significantly improves the accuracy of LLMs at ultra-low bits, e.g., 2-bit LLaMA-7B achieves a 5.5-times memory-saving than original model on NVIDIA A800 GPUs, and 48% decrease of perplexity compared to the state-of-the-art gradient-free PTQ method. Moreover, SliM-LLM+, which is integrated from the extension of SliM-LLM with gradient-based quantizers, further reduces perplexity by 35.1%.

Near-Ideal M-ary LDGM Quantization with Recovery

Analytical Framework of LDGM-based Iterative Quantization with Decimation.

Approaching 1.53-Db Shaping Gain with LDGM Quantization Codes

LQER: Low-Rank Quantization Error Reconstruction for LLMs

A quantization schema with negligible degradation for LDPC decoder.

QuantEase: Optimization-based Quantization for Language Models - An Efficient and Intuitive Algorithm

Adaptive Quantized and Normalized MSA and Its Application for DTMB-A LDPC Codes

Optimizing Gain Codebook of LD-CELP.

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

DB-LLM: Accurate Dual-Binarization for Efficient LLMs

Generalized Mutual Information-Maximizing Quantized Decoding of LDPC Codes With Layered Scheduling

Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM

Min-Sum Approximation Decoding Of Ldpc Codes With Adaptive Non-Uniform Quantization

VARIABLE NON-UNIFORM QUANTIZED BELIEF PROPAGATION ALGORITHM FOR LDPC DECODING

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

The Uniqueness of LLaMA3-70B Series with Per-Channel Quantization

CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization

Extreme Compression of Large Language Models via Additive Quantization

VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models

QERA: an Analytical Framework for Quantization Error Reconstruction