Abstract:Large language models (LLMs) have exhibited exciting progress in multiple scenarios, while the huge computational demands hinder their deployments in lots of real-world applications. As an effective means to reduce memory footprint and inference cost, quantization also faces challenges in performance degradation at low bit-widths. Understanding the impact of quantization on LLM capabilities, especially the generalization ability, is crucial. However, the community's main focus remains on the algorithms and models of quantization, with insufficient attention given to whether the quantized models can retain the strong generalization abilities of LLMs. In this work, we fill this gap by providing a comprehensive benchmark suite for this research topic, including an evaluation system, detailed analyses, and a general toolbox. Specifically, based on the dominant pipeline in LLM quantization, we primarily explore the impact of calibration data distribution on the generalization of quantized LLMs and conduct the benchmark using more than 40 datasets within two main scenarios. Based on this benchmark, we conduct extensive experiments with two well-known LLMs (English and Chinese) and four quantization algorithms to investigate this topic in-depth, yielding several counter-intuitive and valuable findings, e.g., models quantized using a calibration set with the same distribution as the test data are not necessarily optimal. Besides, to facilitate future research, we also release a modular-designed toolbox, which decouples the overall pipeline into several separate components, e.g., base LLM module, dataset module, quantizer module, etc. and allows subsequent researchers to easily assemble their methods through a simple configuration. Our benchmark suite is publicly available at <a class="link-external link-https" href="https://github.com/TsingmaoAI/MI-optimize" rel="external noopener nofollow">this https URL</a>

Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners

Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other

A Comprehensive Evaluation of Quantization Strategies for Large Language Models

On the Compressibility of Quantized Large Language Models

A Comprehensive Study on Quantization Techniques for Large Language Models

LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit

QuantEase: Optimization-based Quantization for Language Models - An Efficient and Intuitive Algorithm

Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study

LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

Post Training Quantization of Large Language Models with Microscaling Formats

Evaluating Quantized Large Language Models

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation

INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation

What Makes Quantization for Large Language Models Hard? an Empirical Study from the Lens of Perturbation

LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment

Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens

Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

How Does Quantization Affect Multilingual LLMs?

"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization