Abstract:Large Language Models (LLMs) have revolutionized natural language processing tasks. However, their practical application is constrained by substantial memory and computational demands. Post-training quantization (PTQ) is considered an effective method to accelerate LLM inference. Despite its growing popularity in LLM model compression, PTQ deployment faces two major challenges. First, low-bit quantization leads to performance degradation. Second, restricted by the limited integer computing unit type on GPUs, quantized matrix operations with different precisions cannot be effectively accelerated. To address these issues, we introduce a novel arbitrary-bit quantization algorithm and inference framework, ABQ-LLM. It achieves superior performance across various quantization settings and enables efficient arbitrary-precision quantized inference on the GPU. ABQ-LLM introduces several key innovations: (1) a distribution correction method for transformer blocks to mitigate distribution differences caused by full quantization of weights and activations, improving performance at low bit-widths. (2) the bit balance strategy to counteract performance degradation from asymmetric distribution issues at very low bit-widths (e.g., 2-bit). (3) an innovative quantization acceleration framework that reconstructs the quantization matrix multiplication of arbitrary precision combinations based on BTC (Binary TensorCore) equivalents, gets rid of the limitations of INT4/INT8 computing units. ABQ-LLM can convert each component bit width gain into actual acceleration gain, maximizing performance under mixed precision(e.g., W6A6, W2A8). Based on W2*A8 quantization configuration on LLaMA-7B model, it achieved a WikiText2 perplexity of 7.59 (2.17$\downarrow $ vs 9.76 in AffineQuant). Compared to SmoothQuant, we realized 1.6$\times$ acceleration improvement and 2.7$\times$ memory compression gain.

Unleashing the Potential of LLMs for Quantum Computing: A Study in Quantum Architecture Design

GPT on a Quantum Computer

Application of Large Language Models to Quantum State Simulation

Quantum Many-Body Physics Calculations with Large Language Models

Comparative Study of Large Language Model Architectures on Frontier

Quantum Large Language Models via Tensor Network Disentanglers

Developing Large Language Models for Quantum Chemistry Simulation Input Generation

Exploring the Potential of Large Language Models in Molecular Tasks: An Insightful Evaluation with GPT‐4

Implementation Guidelines and Innovations in Quantum LSTM Networks

Understanding LLMs: A Comprehensive Overview from Training to Inference

Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code

Recent Advances for Quantum Neural Networks in Generative Learning

A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning

Quantum Machine Learning Architecture Search via Deep Reinforcement Learning

A Comprehensive Study on Quantization Techniques for Large Language Models

ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models

Enabling Large Language Models to Perform Power System Simulations with Previously Unseen Tools: A Case of Daline

Adapting Pre-trained Language Models for Quantum Natural Language Processing

Exploring the Impact of Quantum Computing on Machine Learning Performance

Quantum linear algebra is all you need for Transformer architectures

Extending the reach of quantum computing for materials science with machine learning potentials