A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

Ruihao Gong,Yifu Ding,Zining Wang,Chengtao Lv,Xingyu Zheng,Jinyang Du,Haotong Qin,Jinyang Guo,Michele Magno,Xianglong Liu

2024-09-30

Abstract:Large language models (LLMs) have achieved remarkable advancements in natural language processing, showcasing exceptional performance across various tasks. However, the expensive memory and computational requirements present significant challenges for their practical deployment. Low-bit quantization has emerged as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients, thus decreasing memory usage and computational demands. This paper presents a comprehensive survey of low-bit quantization methods tailored for LLMs, covering the fundamental principles, system implementations, and algorithmic strategies. An overview of basic concepts and new data formats specific to low-bit LLMs is first introduced, followed by a review of frameworks and systems that facilitate low-bit LLMs across various hardware platforms. Then, we categorize and analyze techniques and toolkits for efficient low-bit training and inference of LLMs. Finally, we conclude with a discussion of future trends and potential advancements of low-bit LLMs. Our systematic overview from basic, system, and algorithm perspectives can offer valuable insights and guidelines for future works to enhance the efficiency and applicability of LLMs through low-bit quantization.

Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the high memory and computational demands faced by large language models (LLMs) in practical deployment. Specifically:

1. **High Demand for Memory and Computational Resources**:
- Although large language models perform excellently in natural language processing tasks, their massive number of parameters leads to extremely high memory and computational resource consumption.
- These high demands make it difficult to deploy these models in scenarios with limited resources or where high concurrency is required.

2. **Need for Low-Bit Quantization**:
- Low-bit quantization can significantly reduce memory usage and computational demands by decreasing the bit width of model parameters, activations, and gradients.
- Low-bit quantization not only accelerates inference and training but also reduces storage requirements while maintaining acceptable accuracy.

3. **Lack of System and Algorithm Support**:
- Currently, despite some research and practice in low-bit quantization, there is still insufficient support at the system and algorithm levels.
- A comprehensive overview is needed, covering the fundamental principles of low-bit quantization, system implementation, and algorithm strategies to guide future research and applications.

### Main Contributions of the Paper

1. **Fundamental Principles**:
- Introduce the basic concepts of low-bit quantization, including new low-bit data formats and quantization granularity.

2. **System Implementation**:
- Review frameworks and systems on various hardware platforms that support low-bit LLMs.

3. **Algorithm Strategies**:
- Classify and analyze techniques and toolkits for low-bit training and inference.
- Discuss future trends and directions, including emerging research areas, potential breakthroughs, and the impact of new technologies on LLM quantization.

Through these contents, the paper aims to provide valuable insights and guidance for improving the efficiency and applicability of LLMs.