LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Zechun Liu,Barlas Oguz,Changsheng Zhao,Ernie Chang,Pierre Stock,Yashar Mehdad,Yangyang Shi,Raghuraman Krishnamoorthi,Vikas Chandra

2023-05-29

Abstract:Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to implement more efficient quantization techniques in large - language models (LLMs), especially quantization at low - bit precision. The performance of existing post - training quantization methods drops significantly when reduced to below 8 bits, which limits the application of these models in resource - constrained environments. The paper proposes quantization - aware training for LLMs (LLM - QAT), aiming to conduct quantization training without the original training data through a data - free knowledge distillation method. In addition, this research also pays special attention to the quantization of key - value cache (KV cache), which is crucial for improving the throughput of long - sequence generation tasks.

Specifically, the main contributions of the paper include:
1. **Data - free knowledge distillation**: A data - free method is proposed to generate data for knowledge distillation, enabling quantization training without the original training data.
2. **Quantization of key - value cache**: For the first time, quantization - aware training is applied to the key - value cache, which is very important for improving the efficiency of the model when dealing with long - sequence tasks.
3. **Low - bit quantization**: It shows how to effectively quantize large - language models at 4 - bit precision while maintaining high performance.

Through these methods, the paper shows significant performance improvements on multiple benchmarks, especially in low - bit quantization settings. Compared with traditional post - training quantization methods, LLM - QAT can better preserve the performance of the model.