Abstract:Recently, considerable efforts have been directed towards compressing Large Language Models (LLMs), which showcase groundbreaking capabilities across diverse applications but entail significant deployment costs due to their large sizes. Meanwhile, much less attention has been given to mitigating the costs associated with deploying multiple LLMs of varying sizes despite its practical significance. Thus, this paper introduces \emph{any-precision LLM}, extending the concept of any-precision DNN to LLMs. Addressing challenges in any-precision LLM, we propose a lightweight method for any-precision quantization of LLMs, leveraging a post-training quantization framework, and develop a specialized software engine for its efficient serving. As a result, our solution significantly reduces the high costs of deploying multiple, different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such as 3, 4, ..., $n$ bits, into a memory footprint comparable to a single $n$-bit LLM. All the supported LLMs with varying bit-widths demonstrate state-of-the-art model quality and inference throughput, proving itself to be a compelling option for deployment of multiple, different-sized LLMs. Our code is open-sourced and available online.

What problem does this paper attempt to address?

The paper primarily addresses the cost issues faced when deploying large language models (LLMs), especially the cost challenges when multiple LLMs of different sizes need to be deployed. Specifically, the paper tackles the following key issues: 1. **Reducing the cost of deploying multiple LLMs of different sizes**: - Current research focuses more on compressing a single large language model to reduce its deployment cost, but there is less exploration on how to effectively reduce the cost of deploying multiple LLMs of different sizes. - In practical application scenarios, it may be necessary to dynamically select different LLMs based on different latency requirements, which increases the complexity and cost of deployment. 2. **Proposing the concept of Any-Precision LLM**: - Any-Precision LLM is an extension of the Any-Precision DNN concept, aiming to achieve low-cost deployment of multiple LLMs of different sizes by storing a larger LLM (e.g., an n-bit quantized model) and generating smaller models of different bit widths from it. - This approach can significantly reduce memory usage and avoid the additional overhead of training multiple models. 3. **Addressing the challenges in implementing Any-Precision LLM**: - A lightweight method is needed to achieve Any-Precision quantization, as existing methods typically require training models from scratch, which is impractical for LLMs. - New GPU kernels need to be developed to support quantized matrix-vector multiplication, so that the reduction in bit width can be translated into shorter inference times. 4. **Proposed solutions**: - An incremental upgrade-based Any-Precision quantization method is proposed, which uses a post-training quantization framework to generate low-bit-width models and gradually extends them to higher bit widths while maintaining the Any-Precision property. - A specialized software engine has been developed to optimize the memory layout of weights, reducing memory bandwidth consumption when serving Any-Precision LLMs. In summary, the paper aims to address the memory and training cost issues faced when deploying multiple LLMs of different sizes through Any-Precision LLM, thereby achieving an efficient and low-cost deployment solution.

Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

HotaQ: Hardware Oriented Token Adaptive Quantization for Large Language Models

SqueezeLLM: Dense-and-Sparse Quantization

One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments

Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

Progressive Mixed-Precision Decoding for Efficient LLM Inference

Channel-Wise Mixed-Precision Quantization for Large Language Models

LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models

LQER: Low-Rank Quantization Error Reconstruction for LLMs

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models

Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization

Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?

I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models

CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs

DB-LLM: Accurate Dual-Binarization for Efficient LLMs

Aggressive Post-Training Compression on Extremely Large Language Models

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference