Abstract:Large language models (LLMs) have demonstrated impressive abilities in various domains while the inference cost is expensive. Many previous studies exploit quantization methods to reduce LLM inference cost by reducing latency and memory consumption. Applying 2-bit single-precision weight quantization brings >3% accuracy loss, so the state-of-the-art methods use mixed-precision methods for LLMs (e.g. Llama2-7b, etc.) to improve the accuracy. However, challenges still exist: (1) Uneven distribution in weight matrix. (2) Large speed degradation by adding sparse outliers. (3) Time-consuming dequantization operations on GPUs. To tackle these challenges and enable fast and efficient LLM inference on GPUs, we propose the following techniques in this paper. (1) Intra-weight mixed-precision quantization. (2) Exclusive 2-bit sparse outlier with minimum speed degradation. (3) Asynchronous dequantization. We conduct extensive experiments on different model families (e.g. Llama3, etc.) and model sizes. We achieve 2.91-bit for each weight considering all scales/zeros for different models with negligible loss. As a result, with our 2/4/16 mixed-precision quantization for each weight matrix and asynchronous dequantization during inference, our design achieves an end-to-end speedup for Llama2-7b is 1.74x over the original model, and we reduce both runtime cost and total cost by up to 2.53x and 2.29x with less GPU requirements.

What problem does this paper attempt to address?

This paper attempts to solve the high - cost problem in the reasoning process of large - language models (LLMs). Specifically, it mainly faces the following three challenges: 1. **Uneven distribution in the weight matrix**: Weights are quantized in groups, but the weight ranges within some groups are large. Previous methods adopted mixed - precision quantization between groups, ignoring the range differences between groups within each weight matrix, resulting in an accuracy loss of more than 2.7%. 2. **Large speed drop caused by adding sparse outliers**: Retaining sparse outliers can improve accuracy, but it will affect the speed according to the proportion of outliers. For example, 1.5% of outliers will cause a speed drop of more than 30%. 3. **Time - consuming de - quantization operations on GPU**: Mainstream methods need to perform de - quantization operations to execute calculations on the quantized weights. Since the scale of the groups is also quantized, second - order de - quantization operations are applied. These de - quantization operations lead to more than 50% of the execution time. To solve these problems, the paper proposes the following techniques: 1. **Mixed - precision quantization within groups**: Only use 4 - bit quantization for a small number of groups with higher sensitivity (larger Hessian values and range changes), and consider memory alignment on GPU. 2. **Exclusive 2 - bit sparse outliers with minimum speed drop**: Only retain a small part (< 0.5%) of larger weights in 2 - bit groups as sparse outliers to reduce the average bit increment and improve speed. 3. **Asynchronous de - quantization**: It is pointed out that calculating the scale of each group in second - order de - quantization is independent of loading the weights of each group in first - order de - quantization. Therefore, an asynchronous de - quantization method on GPU is designed. Through these techniques, the paper has carried out extensive experiments on different model families (such as Llama3, etc.) and model sizes, achieving a quantization effect of 2.91 bits per weight and improving the reasoning speed with almost no loss. For example, for the Llama2 - 7b model, the end - to - end speed is increased by 1.74 times, the running time and total cost are reduced by up to 2.53 times and 2.29 times respectively, while reducing the GPU requirements.

Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization

HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language Models

ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

SqueezeLLM: Dense-and-Sparse Quantization

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

Progressive Mixed-Precision Decoding for Efficient LLM Inference

SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM

A Speed Odyssey for Deployable Quantization of LLMs

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems

FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM

QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

MixPE: Quantization and Hardware Co-design for Efficient LLM Inference

VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models

"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

Quantization and Hardware Architecture Co-Design for Matrix-Vector Multiplications of Large Language Models