Abstract:The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significant roadblock to large-scale training. For instance, a 7B model typically requires at least 60 GB of GPU memory with full parameter training, which presents challenges for researchers without access to high-resource environments. Parameter Efficient Fine-Tuning techniques such as Low-Rank Adaptation (LoRA) have been proposed to alleviate this problem. However, in most large-scale fine-tuning settings, their performance does not reach the level of full parameter training because they confine the parameter search to a low-rank subspace. Attempting to complement this deficiency, we investigate the layerwise properties of LoRA on fine-tuning tasks and observe an unexpected but consistent skewness of weight norms across different layers. Utilizing this key observation, a surprisingly simple training strategy is discovered, which outperforms both LoRA and full parameter training in a wide range of settings with memory costs as low as LoRA. We name it Layerwise Importance Sampled AdamW (LISA), a promising alternative for LoRA, which applies the idea of importance sampling to different layers in LLMs and randomly freezes most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning in downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over 10%-35% in terms of MT-Bench score while achieving on-par or better performance in MMLU, AGIEval and WinoGrande. On large models, specifically LLaMA-2-70B, LISA surpasses LoRA on MT-Bench, GSM8K, and PubMedQA, demonstrating its effectiveness across different domains.

What problem does this paper attempt to address?

The paper primarily addresses the issue of memory consumption during the fine-tuning process of large language models (LLMs) and proposes a new method—Layerwise Importance Sampled AdamW (LISA)—to improve memory efficiency while maintaining or enhancing model performance. The paper points out that although LLMs have made significant progress in various tasks, their massive memory consumption has become a major obstacle for large-scale training. For example, a model with 7B parameters typically requires at least 60GB of GPU memory for full-parameter training. To address this issue, parameter-efficient fine-tuning techniques such as Low-Rank Adaptation (LoRA) have been proposed, but in most large-scale fine-tuning settings, these techniques have not achieved the performance level of full-parameter training. The key contributions of the paper include: 1. **Discovery**: Through experimental studies on LoRA, the authors observed that the weight norm distribution of LoRA across different layers exhibits an abnormal skewness, with the bottom and/or top layers occupying most of the weights, while other self-attention layers occupy only a small portion. This indicates that different layers have different importance during updates. 2. **Proposing LISA**: Based on the above observations, the authors proposed a new algorithm called LISA. LISA leverages the idea of importance sampling by selectively updating key layers in LLMs and freezing other layers, achieving similar or even lower memory consumption compared to LoRA, while demonstrating superior performance to LoRA and even full-parameter training in various settings. 3. **Validating Effectiveness**: Through extensive experiments, the paper demonstrates that LISA outperforms LoRA in fine-tuning tasks of modern LLMs. In multiple benchmarks, LISA scores 10%-35% higher than LoRA on MT-Bench and achieves better performance on several evaluation metrics such as MMLU, AGIEval, and WinoGrande. Additionally, LISA has shown stability and effectiveness across different scales and tasks, including instruction following, medical question answering, and mathematical problems. In summary, LISA provides a potential improvement over LoRA, especially when dealing with large language models, significantly reducing memory consumption while ensuring performance.

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation

Full Parameter Fine-tuning for Large Language Models with Limited Resources

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

LoRA ensembles for large language model fine-tuning

AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning

FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models

Sparse Matrix in Large Language Model Fine-tuning