Abstract:The Large Language Models (LLMs) have been popular and widely used in creative ways because of their powerful capabilities. However, the substantial model size and complexity prevent LLMs from being implemented on resourceconstrained computing devices efficiently. Recent works that utilize quantization to boost the on-device efficiency of LLMs show that 8-bit or lower weight quantization is feasible with minimal impact on task performance with full-precision activation. Nevertheless, weight-only quantization does not leverage the potential acceleration on general edge processors, which typically support 16x16 and 8x8 integer multipliers. Additionally, it cannot benefit from specialized devices like Field-Programmable Gate Arrays (FPGAs), which offer reconfigure computing to design custom multipliers of arbitrary bit width. In this paper, we propose HotaQ, a Hardware-oriented token adaptive Quantization framework for general LLMs, and implement end-to-end accelerators on multiple devices. Considering the hardware profiling and activation analysis, we first introduce a basic activation quantization strategy to balance the trade-off of task performance and real inference speed. Then we integrate the activation-aware token pruning workflow to reduce the outliers and the adverse impact on attentivity. For mobile devices, we notably design the SIMD-based 4-bit multiplier and the efficient TRIP matrix multiplication for mobile devices. For FPGAs, we specially utilize the DSP packing techniques for 4/8-bit systolic-array-based multipliers and introduce an analytical model for achieving the best performance with limited on-chip resources. We apply our framework on different scales of LLMs, including LLaMA, OPT, and BLOOM with 4/8-bit for the activation and 4-bit for the weight. Experiments show that HotaQ achieves quantization on both model weights and activations while maintaining task performance comparable to existing weight-only quantization methods or even FP16 models. In the 4/8-bit scenario, HotaQ achieves an on-device speedup of up to 2.5× and 5.2× compared to its FP16 counterparts across edge devices and FPGAs respectively, marking a pioneering advancement in this domain.

Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other

Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners

HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language Models

A Comprehensive Evaluation of Quantization Strategies for Large Language Models

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

Post Training Quantization of Large Language Models with Microscaling Formats

VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

LQER: Low-Rank Quantization Error Reconstruction for LLMs

Scaling laws for post-training quantized large language models

Understanding the difficulty of low-precision post-training quantization of large language models

Foundations of Large Language Model Compression -- Part 1: Weight Quantization

One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments

WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More

AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models

Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens

What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

When Quantization Affects Confidence of Large Language Models?