Abstract:The significant resource requirements associated with Large-scale Language Models (LLMs) have generated considerable interest in the development of techniques aimed at compressing and accelerating neural networks. Among these techniques, Post-Training Quantization (PTQ) has emerged as a subject of considerable interest due to its noteworthy compression efficiency and cost-effectiveness in the context of training. Existing PTQ methods for LLMs limit the optimization scope to scaling transformations between pre- and post-quantization weights. In this paper, we advocate for the direct optimization using equivalent Affine transformations in PTQ (AffineQuant). This approach extends the optimization scope and thus significantly minimizing quantization errors. Additionally, by employing the corresponding inverse matrix, we can ensure equivalence between the pre- and post-quantization outputs of PTQ, thereby maintaining its efficiency and generalization capabilities. To ensure the invertibility of the transformation during optimization, we further introduce a gradual mask optimization method. This method initially focuses on optimizing the diagonal elements and gradually extends to the other elements. Such an approach aligns with the Levy-Desplanques theorem, theoretically ensuring invertibility of the transformation. As a result, significant performance improvements are evident across different LLMs on diverse datasets. To illustrate, we attain a C4 perplexity of 15.76 (2.26 lower vs 18.02 in OmniQuant) on the LLaMA2-7B model of W4A4 quantization without overhead. On zero-shot tasks, AffineQuant achieves an average of 58.61 accuracy (1.98 lower vs 56.63 in OmniQuant) when using 4/4-bit quantization for LLaMA-30B, which setting a new state-of-the-art benchmark for PTQ in LLMs.

Online Optimized Product Quantization

Online Additive Quantization

Differentiable Optimized Product Quantization and Beyond

Online Variable Coding Length Product Quantization for Fast Nearest Neighbor Search in Mobile Retrieval.

Pse: Mixed Quantization Framework of Neural Networks for Efficient Deployment

Local Deep Learning Quantization for Approximate Nearest Neighbor Search

Transformed Residual Quantization for Approximate Nearest Neighbor Search

Iteratively multiple projections optimization for product quantization in nearest neighbor search

Flexible Product Quantization for Fast Approximate Nearest Neighbor Search.

Codebook-softened product quantization for high accuracy approximate nearest neighbor search

Product quantization with dual codebooks for approximate nearest neighbor search

Mean-Removed Product Quantization for Approximate Nearest Neighbor Search.

Online Residual Quantization Via Streaming Data Correlation Preserving

Deep Product Quantization Module for Efficient Image Retrieval

Anisotropic Additive Quantization for Fast Inner Product Search

Progressive Similarity Preservation Learning for Deep Scalable Product Quantization

Scalable Image Retrieval by Sparse Product Quantization

Bilinear Optimized Product Quantization for Scalable Visual Content Analysis.

Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective

Matching-oriented Embedding Quantization for Ad-hoc Retrieval.

AffineQuant: Affine Transformation Quantization for Large Language Models