Abstract:Large Language Models (LLMs) excel in NLP, but their demands hinder their widespread deployment. While Quantization-Aware Training (QAT) offers a solution, its extensive training costs make Post-Training Quantization (PTQ) a more practical approach for LLMs. In existing studies, activation outliers in particular channels are identified as the bottleneck to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in a severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassembly and channel assembly, which first breaks down the outlier channels into several sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for efficiency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further compensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns a small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM can obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the previous state-of-the-art method by 7.89% on the average accuracy across five zero-shot tasks.

Median Based Adaptive Quantization of Log-Likelihood Ratios

Asymptotically Optimal One-Bit Quantizer Design for Weak-Signal Detection in Generalized Gaussian Noise and Lossy Binary Communication Channel.

Analytical Framework of LDGM-based Iterative Quantization with Decimation.

A log-likelihood ratio nonuniform quantization scheme for cooperative spectrum sensing

Low-Complexity Algorithm for Log Likelihood Ratios in Coded MIMO-OFDM Communications

Distributed Adaptive Quantization for Wireless Sensor Networks: from Delta Modulation to Maximum Likelihood

CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs

Adaptive Quantization for Key Generation in Low-Power Wide-Area Networks

AIML-Enhanced Adaptive Quantization for Channel Sounding Feedback in Wireless Communications

On Bit Allocation and Lagrange Multiplier Adjustment for Rate-Distortion Optimized H.264 Rate Control

Min-Sum Approximation Decoding Of Ldpc Codes With Adaptive Non-Uniform Quantization

Maximum Average Mutual Information Quantisation of Low Density Parity Check Codes for Eight-Phase-Shift Keying Modulation

Online variable-length source coding for minimum bitrate LQG control

Study of Joint Automatic Gain Control and MMSE Receiver Design Techniques for Quantized Multiuser Multiple-Antenna Systems

QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models

Optimality of the Logarithmic Quantizer for Stabilization of Linear Systems: Achieving the Minimum Data Rate

On Transmission Model for Massive MIMO under Low-Resolution Output Quantization

Physical Layer Secret Key Agreement Using One-Bit Quantization and Low-Density Parity-Check Codes

Method to reduce the signal-to-noise ratio required for modulation recognition based on logarithmic properties.

Non-Uniform Quantization Scheme For The Decoding Of Low-Density Parity-Check Codes With The Sum-Product Algorithm

On the Design of Quantization Functions for Uplink Massive MIMO with Low-Resolution ADCs