LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models

Yaowei Zheng,Richong Zhang,Junhao Zhang,Yanhan Ye,Zheyan Luo,Zhangchi Feng,Yongqiang Ma

2024-06-28

Abstract:Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, it requires non-trivial efforts to implement these methods on different models. We present LlamaFactory, a unified framework that integrates a suite of cutting-edge efficient training methods. It provides a solution for flexibly customizing the fine-tuning of 100+ LLMs without the need for coding through the built-in web UI LlamaBoard. We empirically validate the efficiency and effectiveness of our framework on language modeling and text generation tasks. It has been released at <a class="link-external link-https" href="https://github.com/hiyouga/LLaMA-Factory" rel="external noopener nofollow">this https URL</a> and received over 25,000 stars and 3,000 forks.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the challenge of efficient fine-tuning faced by large language models (LLMs) when adapting to downstream tasks. Specifically, the paper points out:

1. **Fine-tuning challenges under resource constraints**: Due to the large number of parameters in LLMs, direct fine-tuning requires a significant amount of computational resources, which is a major obstacle for many researchers and developers.
2. **Lack of a systematic framework**: Although the community has proposed various efficient fine-tuning methods, there is a lack of uniformity and systematicity among these methods, making it difficult for users to flexibly choose and apply them.

To address these issues, the paper proposes a unified framework called **LLAMA FACTORY**, which integrates multiple advanced efficient training methods, aiming to simplify and optimize the fine-tuning process of LLMs through the following points:

- **Modular design**: By adopting a modular design, it reduces the dependencies between different models, datasets, and training methods, allowing the framework to flexibly adapt to hundreds of different LLMs.
- **User-friendly interface**: Provides a web-based user interface **LLAMA BOARD**, through which users can easily configure and launch fine-tuning tasks without writing code.
- **Efficient fine-tuning techniques**: Integrates various efficient fine-tuning techniques, such as Freeze-tuning, GaLore (low-rank projection), and LoRA (low-rank adapters), significantly improving the efficiency and effectiveness of fine-tuning.

Through these designs, **LLAMA FACTORY** aims to lower the threshold for fine-tuning LLMs, enabling more researchers and developers to conveniently utilize these powerful models for various downstream tasks.