Abstract:Large Language Models (LLMs) have demonstrated impressive performance across various domains. However, the enormous number of model parameters makes fine-tuning challenging, significantly limiting their application and deployment. Existing solutions combine parameter quantization with Low-Rank Adaptation (LoRA), greatly reducing memory usage but resulting in noticeable performance degradation. In this paper, we identify an imbalance in fine-tuning quantized pre-trained models: overly complex adapter inputs and outputs versus low effective trainability of the adaptation. We propose Quantized LLMs with Balanced-rank Adaptation (Q-BaRA), which simplifies the adapter inputs and outputs while increasing the adapter's rank to achieve a more suitable balance for fine-tuning quantized LLMs. Additionally, for scenarios where fine-tuned LLMs need to be deployed as low-precision inference models, we introduce Quantization-Aware Fine-tuning with Higher Rank Adaptation (QA-HiRA), which simplifies the adapter inputs and outputs to align with the pre-trained model's block-wise quantization while employing a single matrix to achieve a higher rank. Both Q-BaRA and QA-HiRA are easily implemented and offer the following optimizations: (i) Q-BaRA consistently achieves the highest accuracy compared to baselines and other variants, requiring the same number of trainable parameters and computational effort; (ii) QA-HiRA naturally merges adapter parameters into the block-wise quantized model after fine-tuning, achieving the highest accuracy compared to other methods. We apply our Q-BaRA and QA-HiRA to the LLaMA and LLaMA2 model families and validate their effectiveness across different fine-tuning datasets and downstream scenarios. Code will be made available at \href{<a class="link-external link-https" href="https://github.com/xiaocaigou/qbaraqahira" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/xiaocaigou/qbaraqahira" rel="external noopener nofollow">this https URL</a>}

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

Bayesian Low-rank Adaptation for Large Language Models

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

Hyperparameter Optimization for Large Language Model Instruction-Tuning

Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation

Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models

LoRTA: Low Rank Tensor Adaptation of Large Language Models

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation

LoRA ensembles for large language model fine-tuning

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance

BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-task Large Language Models

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning