Abstract:Large language models (LLMs) achieve remarkable performance in natural language understanding but require substantial computation and memory resources. Post-training quantization (PTQ) is a powerful compression technique extensively investigated in LLMs. However, existing PTQ methods are still not ideal in terms of accuracy and efficiency, especially with below 4 bit-widths. Standard PTQ methods using group-wise quantization suffer difficulties in quantizing LLMs accurately to such low-bit, but advanced methods remaining high-precision weights element-wisely are hard to realize their theoretical hardware efficiency. This paper presents a Salience-Driven Mixed-Precision Quantization scheme for LLMs, namely SliM-LLM. The scheme exploits the salience distribution of weights to determine optimal bit-width and quantizers for accurate LLM quantization, while aligning bit-width partition to groups for compact memory usage and fast integer inference. Specifically, the proposed SliM-LLM mainly relies on two novel techniques: (1) Salience-Determined Bit Allocation utilizes the clustering characteristics of salience distribution to allocate the bit-widths of each group, increasing the accuracy of quantized LLMs and maintaining the inference efficiency; (2) Salience-Weighted Quantizer Calibration optimizes the parameters of the quantizer by considering the element-wise salience within the group, balancing the maintenance of salient information and minimization of errors. Comprehensive experiments show that SliM-LLM significantly improves the accuracy of LLMs at ultra-low bits, e.g., 2-bit LLaMA-7B achieves a 5.5-times memory-saving than original model on NVIDIA A800 GPUs, and 48% decrease of perplexity compared to the state-of-the-art gradient-free PTQ method. Moreover, SliM-LLM+, which is integrated from the extension of SliM-LLM with gradient-based quantizers, further reduces perplexity by 35.1%.

SDQ: Sparse Decomposed Quantization for LLM Inference

SqueezeLLM: Dense-and-Sparse Quantization

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

On the Compressibility of Quantized Large Language Models

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models

LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit

QuantEase: Optimization-based Quantization for Language Models - An Efficient and Intuitive Algorithm

A Comprehensive Evaluation of Quantization Strategies for Large Language Models

LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

Extreme Compression of Large Language Models via Additive Quantization

SLiM: One-shot Quantized Sparse Plus Low-rank Approximation of LLMs

Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

LQER: Low-Rank Quantization Error Reconstruction for LLMs

Foundations of Large Language Model Compression -- Part 1: Weight Quantization

Post Training Quantization of Large Language Models with Microscaling Formats

Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization