Abstract:Large language models (LLMs) are omnipresent, however their practical deployment is challenging due to their ever increasing computational and memory demands. Quantization is one of the most effective ways to make them more compute and memory efficient. Quantization-aware training (QAT) methods, generally produce the best quantized performance, however it comes at the cost of potentially long training time and excessive memory usage, making it impractical when applying for LLMs. Inspired by parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) literature, we propose LR-QAT -- a lightweight and memory-efficient QAT algorithm for LLMs. LR-QAT employs several components to save memory without sacrificing predictive performance: (a) low-rank auxiliary weights that are aware of the quantization grid; (b) a downcasting operator using fixed-point or double-packed integers and (c) checkpointing. Unlike most related work, our method (i) is inference-efficient, leading to no additional overhead compared to traditional PTQ; (ii) can be seen as a general extended pretraining framework, meaning that the resulting model can still be utilized for any downstream task afterwards; (iii) can be applied across a wide range of quantization settings, such as different choices quantization granularity, activation quantization, and seamlessly combined with many PTQ techniques. We apply LR-QAT to LLaMA-1/2/3 and Mistral model families and validate its effectiveness on several downstream tasks. Our method outperforms common post-training quantization (PTQ) approaches and reaches the same model performance as full-model QAT at the fraction of its memory usage. Specifically, we can train a 7B LLM on a single consumer grade GPU with 24GB of memory. Our source code is available at <a class="link-external link-https" href="https://github.com/qualcomm-ai-research/LR-QAT" rel="external noopener nofollow">this https URL</a>

L3iTC at the FinLLM Challenge Task: Quantization for Financial Text Classification & Summarization

CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications

FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation

'Finance Wizard' at the FinLLM Challenge Task: Financial Text Summarization

Low-Rank Quantization-Aware Training for LLMs

ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs

CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs

How Good Are Low-bit Quantized LLaMA3 Models? an Empirical Study

Evaluating Quantized Large Language Models

QuantEase: Optimization-based Quantization for Language Models - An Efficient and Intuitive Algorithm

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models

Parameter-Efficient Instruction Tuning of Large Language Models For Extreme Financial Numeral Labelling

ApiQ: Finetuning of 2-Bit Quantized Large Language Model

AffineQuant: Affine Transformation Quantization for Large Language Models

Dólares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English