Abstract:Large language models (LLMs) have exhibited exciting progress in multiple scenarios, while the huge computational demands hinder their deployments in lots of real-world applications. As an effective means to reduce memory footprint and inference cost, quantization also faces challenges in performance degradation at low bit-widths. Understanding the impact of quantization on LLM capabilities, especially the generalization ability, is crucial. However, the community's main focus remains on the algorithms and models of quantization, with insufficient attention given to whether the quantized models can retain the strong generalization abilities of LLMs. In this work, we fill this gap by providing a comprehensive benchmark suite for this research topic, including an evaluation system, detailed analyses, and a general toolbox. Specifically, based on the dominant pipeline in LLM quantization, we primarily explore the impact of calibration data distribution on the generalization of quantized LLMs and conduct the benchmark using more than 40 datasets within two main scenarios. Based on this benchmark, we conduct extensive experiments with two well-known LLMs (English and Chinese) and four quantization algorithms to investigate this topic in-depth, yielding several counter-intuitive and valuable findings, e.g., models quantized using a calibration set with the same distribution as the test data are not necessarily optimal. Besides, to facilitate future research, we also release a modular-designed toolbox, which decouples the overall pipeline into several separate components, e.g., base LLM module, dataset module, quantizer module, etc. and allows subsequent researchers to easily assemble their methods through a simple configuration. Our benchmark suite is publicly available at <a class="link-external link-https" href="https://github.com/TsingmaoAI/MI-optimize" rel="external noopener nofollow">this https URL</a>

FunQuant: A R package to perform quantization in the context of rare events and time-consuming simulations

Functional quantization-based stratified sampling methods

Quantization for a set of discrete distributions on the set of natural numbers

Quantizing rare random maps: application to flooding visualization

Augmented quantization: a general approach to mixture models

EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs

QuantEase: Optimization-based Quantization for Language Models - An Efficient and Intuitive Algorithm

Quantum Quantile Mechanics: Solving Stochastic Differential Equations for Generating Time-Series

Quantization for a Mixture of Uniform Distributions Associated with Probability Vectors

ISQuant: apply squant to the real deployment

Quantized Low-Rank Multivariate Regression with Random Dithering

Scalar Quantization as Sparse Least Square Optimization

Automatic Quantization for Physics-Based Simulation

Quantization based recursive Importance Sampling

Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox

Examples of Berezin-Toeplitz Quantization: Finite sets and Unit Interval

FlatQuant: Flatness Matters for LLM Quantization

quant: a minimalist interval method for time series classification

SQuant: On-the-Fly Data-Free Quantization Via Diagonal Hessian Approximation

LLMEasyQuant -- An Easy to Use Toolkit for LLM Quantization

Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels