Abstract:Large language models (LLMs) have exhibited exciting progress in multiple scenarios, while the huge computational demands hinder their deployments in lots of real-world applications. As an effective means to reduce memory footprint and inference cost, quantization also faces challenges in performance degradation at low bit-widths. Understanding the impact of quantization on LLM capabilities, especially the generalization ability, is crucial. However, the community's main focus remains on the algorithms and models of quantization, with insufficient attention given to whether the quantized models can retain the strong generalization abilities of LLMs. In this work, we fill this gap by providing a comprehensive benchmark suite for this research topic, including an evaluation system, detailed analyses, and a general toolbox. Specifically, based on the dominant pipeline in LLM quantization, we primarily explore the impact of calibration data distribution on the generalization of quantized LLMs and conduct the benchmark using more than 40 datasets within two main scenarios. Based on this benchmark, we conduct extensive experiments with two well-known LLMs (English and Chinese) and four quantization algorithms to investigate this topic in-depth, yielding several counter-intuitive and valuable findings, e.g., models quantized using a calibration set with the same distribution as the test data are not necessarily optimal. Besides, to facilitate future research, we also release a modular-designed toolbox, which decouples the overall pipeline into several separate components, e.g., base LLM module, dataset module, quantizer module, etc. and allows subsequent researchers to easily assemble their methods through a simple configuration. Our benchmark suite is publicly available at <a class="link-external link-https" href="https://github.com/TsingmaoAI/MI-optimize" rel="external noopener nofollow">this https URL</a>

The Improved Quantization Rule and the Langer Modification

The Magnificent Realm of Affine Quantization: valid results for particles, fields, and gravity

Quantization of the Hamilton Equations of Motion

A new development of modified quantum mechanics

Quantum phase transition in an atom-molecule conversion system with atomic hopping

Quantization formula for singular reductions

Improved Quantization Method of Coupled Circuits in Charge Discrete Space

On the best lattice quantizers

Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox

Lectures on Non-BPS Dirichlet branes

Canonical Quantization of Classical Fields in Finite Volume

Quantization for a set of discrete distributions on the set of natural numbers

On Modulation and Translation Invariant Operators and the Heisenberg Module

Quantization commutes with reduction again: the quantum GIT conjecture I

Any l-state solutions of the Woods-Saxon potential in arbitrary dimensions within the new improved quantization rule

AffineQuant: Affine Transformation Quantization for Large Language Models

Reformulation of the Covering and Quantizer Problems as Ground States of Interacting Particles

A δ-free approach to quantization of transmission lines connected to lumped circuits

What Makes Quantization for Large Language Models Hard? an Empirical Study from the Lens of Perturbation

Multiple Quantization and the Concept of Information

Quantization of linear acoustic and elastic wave models in characterizations of isomorphism