QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models

Saleh Ashkboos,Ilia Markov,Elias Frantar,Tingxuan Zhong,Xincheng Wang,Jie Ren,Torsten Hoefler,Dan Alistarh
2023-11-02
Abstract:Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. Yet, the vast majority of existing work focuses on weight-only quantization, which can reduce runtime costs in the memory-bound one-token-at-a-time generative setting, but does not address them in compute-bound scenarios, such as batched inference or prompt processing. In this paper, we address the general quantization problem, where both weights and activations should be quantized. We show, for the first time, that the majority of inference computations for large generative models such as LLaMA, OPT, and Falcon can be performed with both weights and activations being cast to 4 bits, in a way that leads to practical speedups, while at the same time maintaining good accuracy. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit, while keeping some outlier weights and activations in higher-precision. The key feature of our scheme is that it is designed with computational efficiency in mind: we provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x relative to FP16 execution. We provide detailed studies for models from the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate inference using quantization plus 2:4 sparsity. Code is available at: <a class="link-external link-https" href="https://github.com/IST-DASLab/QUIK" rel="external noopener nofollow">this https URL</a>.
Machine Learning
What problem does this paper attempt to address?
This paper mainly discusses how to achieve end-to-end 4-bit inference for large language models (LLMs), especially for generative models such as LLaMA, OPT, and Falcon. Most existing work focuses only on weight quantization, which can reduce runtime costs in memory-constrained single-token generation scenarios, but cannot solve the efficiency problem in computation-intensive scenarios such as batch inference or computation in prompt handling. The paper proposes a hybrid quantization strategy called QUIK, which quantizes the majority of weights and activations to 4 bits while keeping a portion of exceptional weights and activations at higher precision. QUIK is designed considering computational efficiency, provides matching GPU kernels, and achieves up to 3.4 times higher actual throughput compared to FP16 execution. With this approach, QuiK significantly improves the computational speed of modern LLMs while maintaining good accuracy and reducing memory requirements. Experimental results show that QuiK achieves speed improvements across different model sizes and even achieves 3.4 times acceleration with minor accuracy loss for sensitive LLaMA-2 models, even with a parameter count of 70 billion. Additionally, QuiK reduces the GPU memory requirements, allowing for fewer GPUs to accurately execute LLMs in FP16 format.