When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

Biao Zhang,Zhongtao Liu,Colin Cherry,Orhan Firat

2024-02-27

Abstract:While large language models (LLMs) often adopt finetuning to unlock their capabilities for downstream applications, our understanding on the inductive biases (especially the scaling properties) of different finetuning methods is still limited. To fill this gap, we conduct systematic experiments studying whether and how different scaling factors, including LLM model size, pretraining data size, new finetuning parameter size and finetuning data size, affect the finetuning performance. We consider two types of finetuning -- full-model tuning (FMT) and parameter efficient tuning (PET, including prompt tuning and LoRA), and explore their scaling behaviors in the data-limited regime where the LLM model size substantially outweighs the finetuning data size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and experiments on bilingual machine translation and multilingual summarization benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative joint scaling law between finetuning data size and each other scaling factor; 2) LLM finetuning benefits more from LLM model scaling than pretraining data scaling, and PET parameter scaling is generally ineffective; and 3) the optimal finetuning method is highly task- and finetuning data-dependent. We hope our findings could shed light on understanding, selecting and developing LLM finetuning methods.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

This paper investigates the effects of fine-tuning large-scale language models (LLMs), specifically studying the impact of different factors such as model size, pre-training data size, fine-tuning parameter size, and fine-tuning data size on the fine-tuning performance. The authors conduct systematic experiments to analyze the scaling behavior of Full Model Tuning (FMT) and Parameter-Efficient Tuning (PET, including prompt tuning and low-rank adaptation) under data-limited conditions. The study reveals the following findings:

1. Fine-tuning LLMs follows a multiplicative joint scaling law based on data size and other factors.
2. Scaling up the model size is more beneficial for LLM fine-tuning compared to scaling up the pre-training data size.
3. Increasing the parameter size does not always lead to improved performance for PET methods (such as LoRA and prompt tuning), especially for prompt tuning, although LoRA exhibits better training stability.
4. The optimal fine-tuning method for LLMs highly depends on the task and fine-tuning data, making it a complex problem to select the best fine-tuning approach.
5. Fine-tuning LLMs can promote zero-shot generalization for related tasks, with PET methods generally outperforming FMT in this respect.

The paper empirically validates these findings and suggests that these observed results contribute to understanding the impact of pre-training factors on fine-tuning, providing insights for LLM pre-training. Additionally, the study emphasizes how to choose appropriate fine-tuning strategies based on task requirements in resource-constrained scenarios.